Encoder vs decoder: encoder sees all tokens bidirectionally, decoder sees only past tokens

Masking (behavior)

Can you not see what's right in front of you?

BPE tokenization does: iteratively merges the most frequent adjacent byte pairs

How do we make computers understand language better?

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

Large language model

LLMs can generate, summarize, translate, and analyze text in many contexts

Greedy vs beam search decoding: greedy picks best token, beam maintains k candidates

Ever wondered why Google Search sometimes shows you the top results first?

the tokenizer's special tokens do: [CLS], [SEP], [PAD], [MASK] have specific roles

[CLS] marks the start of input, [SEP] denotes separation, [PAD] fills space, [MASK] hides words for prediction