self-attention: Attention(Q,K,V) = softmax(QK^T/√d_k)V

Ever wondered how computers understand what's important in a sentence?

Imagine you're reading a long email and trying to find the key points quickly. You want to skip the less important details and focus on what's truly important.

Think of the email as a sequence of words. The attention mechanism helps the computer figure out which words are more important, like picking out the key points in your email.

Example

If the sentence is "The meeting was postponed due to unforeseen circumstances," the attention mechanism might highlight "postponed" as the key word.

Remember this

The key insight is that attention helps computers focus on the most important parts of data, like picking out key points in an email.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Write the attention score formula before softmax: e_ij = a(s_i, h_j)

How do we know what's important in a sentence?

Write the multi-head attention formula: MultiHead(Q,K,V) = Concat(head_1,...,head_h)W^O

Ever wondered how machines understand the importance of words in a sentence?

grouped query attention (GQA) does

GQA shares KV heads across multiple Q heads for efficient parameter usage

multi-query attention (MQA) is

Multi-query attention (MQA) with shared KV head: Q heads share a single KV head for efficient parameter usage

Softmax function

Softmax converts real numbers into a probability distribution

BLEU

Ever wondered how computers know if a translation makes sense?

Swipe through 100 ML concepts daily

Open Pocket Polymath