"Attention Is All You Need" introduced the transformer architecture in 2017
Attention Is All You Need
"Attention Is All You Need" introduced the transformer architecture in 2017
The transformer architecture revolutionized machine learning by introducing a new deep learning model. Authored by eight scientists and engineers at Google, it was initially focused on improving machine translation techniques. The paper's introduction of the transformer marked a significant advancement in artificial intelligence.
Example
The transformer was first tested on English-to-German translation tasks, demonstrating its effectiveness in handling complex language tasks.
Remember this
The introduction of the transformer architecture has had a profound impact on the field of artificial intelligence, leading to advancements in various applications such as language translation, question answering, and multimodal generative AI.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Transformer (deep learning)
Transformers use multi-head attention for contextualizing tokens
Machine learning in bioinformatics
How do Transformers understand what's important in a sentence?
2022 in science
Why do Transformers sometimes seem to 'ignore' irrelevant parts of the input?
the tokenizer's special tokens do: [CLS], [SEP], [PAD], [MASK] have specific roles
[CLS] marks the start of input, [SEP] denotes separation, [PAD] fills space, [MASK] hides words for prediction
Masking (behavior)
Can you not see what's right in front of you?
ring attention does: distributes long sequences across multiple devices
How can a machine understand and generate human language?
Swipe through 100 ML concepts daily
Open Pocket Polymath