Transformers use multi-head attention for contextualizing tokens
Transformer (deep learning)
Transformers use multi-head attention for contextualizing tokens
In transformers, each token is contextualized through a multi-head attention mechanism. This allows the model to focus on different parts of the input sequence simultaneously, enhancing the representation of each token by considering its context.
Example
For a sentence like "The cat sat on the mat," each word token (e.g., "cat," "sat," "mat") is contextualized by considering its relationship with other words in the sentence.
Remember this
Understanding this helps grasp how transformers achieve efficient and effective language modeling.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Machine learning in bioinformatics
How do Transformers understand what's important in a sentence?
2022 in science
Why do Transformers sometimes seem to 'ignore' irrelevant parts of the input?
Pre-LN transformers are easier to train
Pre-LN transformers use residual connections, allowing gradients to flow more smoothly during backpropagation
ring attention does: distributes long sequences across multiple devices
How can a machine understand and generate human language?
Attention Is All You Need
"Attention Is All You Need" introduced the transformer architecture in 2017
the embedding layer does: maps discrete token IDs to dense learned vectors
Embeddings convert token IDs to dense vectors for neural network processing
Swipe through 100 ML concepts daily
Open Pocket Polymath