Write the attention score formula before softmax: e_ij = a(s_i, h_j)

How do we know what's important in a sentence?

Imagine you're reading an email full of information, but you want to quickly find the key points without reading everything word for word.

Think of each word in the email as a person at a party. Some people are more important to the conversation than others. We want to figure out who those important people are so we can focus on them.

Example

In the email, the word "urgent" next to "meeting" highlights the importance of the meeting over other less significant details.

Remember this

The attention score formula helps us identify which words (or people at the party) are most important to focus on.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

self-attention: Attention(Q,K,V) = softmax(QK^T/√d_k)V

Ever wondered how computers understand what's important in a sentence?

Write the multi-head attention formula: MultiHead(Q,K,V) = Concat(head_1,...,head_h)W^O

Ever wondered how machines understand the importance of words in a sentence?

Softmax function

Softmax converts real numbers into a probability distribution

Precision and recall

Precision = Relevant retrieved instances / All retrieved instances

BLEU

Ever wondered how computers know if a translation makes sense?

TF-IDF scoring

TF-IDF = (Term Frequency) * (Inverse Document Frequency)

Swipe through 100 ML concepts daily

Open Pocket Polymath