the tokenizer's special tokens do: [CLS], [SEP], [PAD], [MASK] have specific roles

subword tokenization solves: handles rare words by breaking into known pieces

Subword tokenization solves rare word handling by breaking into known pieces

Masking (behavior)

Can you not see what's right in front of you?

Unigram tokenization does: starts with large vocabulary and prunes using EM

Unigram tokenization starts with a large vocabulary and prunes using EM

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

BPE tokenization does: iteratively merges the most frequent adjacent byte pairs

How do we make computers understand language better?

WordPiece tokenization does: similar to BPE but uses likelihood instead of frequency

WordPiece tokenization splits words into subwords based on token likelihood rather than frequency