ring attention does: distributes long sequences across multiple devices

"Attention Is All You Need" introduced the transformer architecture in 2017

The transformer architecture, introduced in the 2017 paper "Attention Is All You Need," revolutionized machine learning by providing a new deep learning model that leverages the attention mechanism.

Example

The transformer model was initially used for English-to-German translation tasks.

Remember this

Understanding the transformer architecture is crucial for advancements in AI and machine learning.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Attention Is All You Need

"Attention Is All You Need" introduced the transformer architecture in 2017

Transformer (deep learning)

Transformers use multi-head attention for contextualizing tokens

most transformer operations are memory-bound, not compute-bound

Why do computers sometimes get tired?

paged attention (vLLM) improves serving throughput

Paged attention (vLLM) improves serving throughput by reducing latency through non-contiguous KV-cache pages, enabling faster data retrieval

GAT (Graph Attention Network) adds: learned attention weights between neighbors

GATs use learned attention weights between neighbors

Attention (machine learning)

Why do we sometimes zoom in faster on a scene?

Swipe through 100 ML concepts daily

Open Pocket Polymath