WordPiece tokenization does: similar to BPE but uses likelihood instead of frequency

subword tokenization solves: handles rare words by breaking into known pieces

Subword tokenization solves rare word handling by breaking into known pieces

SentencePiece does differently from BPE: operates on raw text including whitespace

SentencePiece tokenizes text without pre-tokenization, preserving whitespace

BPE tokenization does: iteratively merges the most frequent adjacent byte pairs

How do we make computers understand language better?

BPE tokenization does: iteratively merges the most frequent byte pairs

How can we shrink text files without losing meaning?

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

Unigram tokenization does: starts with large vocabulary and prunes using EM

Unigram tokenization starts with a large vocabulary and prunes using EM