Machine learning in bioinformatics

How do Transformers understand what's important in a sentence?

Imagine you're trying to summarize a long article for a friend. You want to focus on the most important parts.

In Transformers, there's a special way to pick out the key parts of a sentence, like highlighting the most important words when summarizing an article.

Example

If the sentence is "The cat sat on the mat," the Transformer might highlight "cat" and "sat" as key parts.

Remember this

Self-attention weighting helps Transformers focus on the most important words in a sentence.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

2022 in science

Why do Transformers sometimes seem to 'ignore' irrelevant parts of the input?

Transformer (deep learning)

Transformers use multi-head attention for contextualizing tokens

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

ring attention does: distributes long sequences across multiple devices

How can a machine understand and generate human language?

Prompt engineering

The GenAI model learns tasks from examples in the prompt

self-attention: Attention(Q,K,V) = softmax(QK^T/√d_k)V

Ever wondered how computers understand what's important in a sentence?

Swipe through 100 ML concepts daily

Open Pocket Polymath