Subword tokenization solves rare word handling by breaking into known pieces
Image: No machine-readable author provided. Flapdragon assumed (based on copyright claims)., Public domain, via Wikimedia Commons
subword tokenization solves: handles rare words by breaking into known pieces
Subword tokenization solves rare word handling by breaking into known pieces
Related concepts
WordPiece tokenization does: similar to BPE but uses likelihood instead of frequency
WordPiece tokenization splits words into subwords based on token likelihood rather than frequency
Unigram tokenization does: starts with large vocabulary and prunes using EM
Unigram tokenization starts with a large vocabulary and prunes using EM
the tokenizer's special tokens do: [CLS], [SEP], [PAD], [MASK] have specific roles
[CLS] marks the start of input, [SEP] denotes separation, [PAD] fills space, [MASK] hides words for prediction
BPE tokenization does: iteratively merges the most frequent adjacent byte pairs
How do we make computers understand language better?
[CLS] pooling does: uses the first token's embedding as the sentence representation
CLS pooling: uses the first token's embedding as the sentence representation
BPE tokenization does: iteratively merges the most frequent byte pairs
How can we shrink text files without losing meaning?
Swipe through more Machine Learning concepts
Open Pocket Polymath