Transformers use multi-head attention for contextualizing tokens
Transformer (deep learning)
Transformers use multi-head attention for contextualizing tokens
In transformers, each token is contextualized through a multi-head attention mechanism. This allows the model to focus on different parts of the input sequence simultaneously, enhancing the representation of each token by considering its context.
Example
For a sentence like "The cat sat on the mat," each word token (e.g., "cat," "sat," "mat") is contextualized by considering its relationship with other words in the sentence.
Remember this
Understanding this helps grasp how transformers achieve efficient and effective language modeling.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
ring attention does: distributes long sequences across multiple devices
"Attention Is All You Need" introduced the transformer architecture in 2017
Pre-LN transformers are easier to train
Pre-LN transformers use residual connections, allowing gradients to flow more smoothly during backpropagation
Attention Is All You Need
"Attention Is All You Need" introduced the transformer architecture in 2017
the embedding layer does: maps discrete token IDs to dense learned vectors
Embeddings convert token IDs to dense vectors for neural network processing
Masking (behavior)
Can you not see what's right in front of you?
transformers use LayerNorm not BatchNorm
LayerNorm normalizes across all features, accommodating variable-length sequences unlike BatchNorm, which relies on fixed-size batches
Swipe through 100 ML concepts daily
Open Pocket Polymath