ring attention does: distributes long sequences across multiple devices

How can a machine understand and generate human language?

Imagine you're trying to translate a long, complex sentence from English to German, but the sentence is too long for your computer to handle at once.

The transformer architecture breaks down the sentence into smaller parts, processing them separately and then combining them to understand the whole sentence. This way, it can handle long sentences more efficiently.

Example

Instead of trying to translate "The quick brown fox jumps over the lazy dog" all at once, the transformer would handle "The quick brown fox" in one part and "jumps over the lazy dog" in another, then combine them to understand the full sentence.

Remember this

The transformer distributes long sequences across multiple devices for better processing efficiency.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

Machine learning in bioinformatics

How do Transformers understand what's important in a sentence?

Large language model

LLMs can generate, summarize, translate, and analyze text in many contexts

2022 in science

Why do Transformers sometimes seem to 'ignore' irrelevant parts of the input?

Transformer (deep learning)

Transformers use multi-head attention for contextualizing tokens

BPE tokenization does: iteratively merges the most frequent adjacent byte pairs

How do we make computers understand language better?

Swipe through 100 ML concepts daily

Open Pocket Polymath