Encoder vs decoder: encoder sees all tokens bidirectionally, decoder sees only past tokens

Masking (behavior)

Can you not see what's right in front of you?

BPE tokenization does: iteratively merges the most frequent adjacent byte pairs

BPE tokenization merges frequent adjacent byte pairs iteratively

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

Large language model

LLMs can generate, summarize, translate, and analyze text in many contexts

BPE tokenization does: iteratively merges the most frequent byte pairs

BPE tokenizes text by merging frequent byte pairs

Greedy vs beam search decoding: greedy picks best token, beam maintains k candidates

Ever wondered why Google Search sometimes shows you the top results first?