"Attention Is All You Need" introduced the transformer architecture in 2017
Attention Is All You Need
"Attention Is All You Need" introduced the transformer architecture in 2017
The transformer architecture revolutionized machine learning by introducing a new deep learning model. Authored by eight scientists and engineers at Google, it was initially focused on improving machine translation techniques. The paper's introduction of the transformer marked a significant advancement in artificial intelligence.
Example
The transformer was first tested on English-to-German translation tasks, demonstrating its effectiveness in handling complex language tasks.
Remember this
The introduction of the transformer architecture has had a profound impact on the field of artificial intelligence, leading to advancements in various applications such as language translation, question answering, and multimodal generative AI.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
ring attention does: distributes long sequences across multiple devices
"Attention Is All You Need" introduced the transformer architecture in 2017
Transformer (deep learning)
Transformers use multi-head attention for contextualizing tokens
the tokenizer's special tokens do: [CLS], [SEP], [PAD], [MASK] have specific roles
[CLS] marks the start of input, [SEP] denotes separation, [PAD] fills space, [MASK] hides words for prediction
Masking (behavior)
Can you not see what's right in front of you?
paged attention (vLLM) improves serving throughput
Paged attention (vLLM) improves serving throughput by reducing latency through non-contiguous KV-cache pages, enabling faster data retrieval
Attention (machine learning)
Why do we sometimes zoom in faster on a scene?
Swipe through 100 ML concepts daily
Open Pocket Polymath