Write the multi-head attention formula: MultiHead(Q,K,V) = Concat(head_1,...,head_h)W^O

Ever wondered how machines understand the importance of words in a sentence?

Imagine you're reading a long email and trying to find the key points quickly. You don't read the whole thing word for word; instead, you focus on the most important parts.

Think of each word in the sentence as a person at a party. Some people are more important to the conversation than others. The machine decides who to 'listen' to more by assigning importance, like deciding who's talking about the party's theme.

Example

In an email, the machine might 'listen' more to the words "meeting," "deadline," and "budget" because they are key points.

Remember this

Multi-head attention helps machines focus on the most important parts of a sentence, like picking out the key points in an email.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

self-attention: Attention(Q,K,V) = softmax(QK^T/√d_k)V

Ever wondered how computers understand what's important in a sentence?

Write the attention score formula before softmax: e_ij = a(s_i, h_j)

How do we know what's important in a sentence?

multi-query attention (MQA) is

Multi-query attention (MQA) with shared KV head: Q heads share a single KV head for efficient parameter usage

grouped query attention (GQA) does

GQA shares KV heads across multiple Q heads for efficient parameter usage

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

Swipe through 100 ML concepts daily

Open Pocket Polymath