GPTLayer

GPT Layer.

Inherits from tensorflow.keras.layers.Layer.

Arguments

__init__ args:

• depth (int): depth of the model (corresponds to embedding size).
• heads (int): number of attention heads.
• ff_nodes (int): size of Dense ReLU layer in Pointwise FF block.
• rate (float): dropout probability. Defaults to 0.1 as in original paper.

call args:

• input_tensor (tf.tensor): input tensor (usually from PositionalEmbedding layer).

Returns

• (tf.tensor): Layer’s output.

Used in tutorial

• Neural Text Generation with a Custom GPT. (Google Colab)