BPE tokenizes text by merging frequent byte pairs
Image: Unknown authorUnknown author, Public domain, via Wikimedia Commons
BPE tokenization does: iteratively merges the most frequent byte pairs
BPE tokenizes text by merging frequent byte pairs
Byte-pair encoding (BPE) is an algorithm that encodes strings of text into smaller strings using a translation table. This table is created by replacing the most frequent pair of bytes with a new byte that wasn't in the initial dataset.
Example
If the most frequent byte pair in a dataset is "AB," BPE would replace "AB" with a new byte, say "X," and update the translation table accordingly.
Remember this
BPE tokenization helps in creating efficient and effective tokenizers for large language models, improving text encoding and processing.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
BPE tokenization does: iteratively merges the most frequent adjacent byte pairs
BPE tokenization merges frequent adjacent byte pairs iteratively
WordPiece tokenization does: similar to BPE but uses likelihood instead of frequency
WordPiece tokenization splits words into subwords based on token likelihood rather than frequency
SentencePiece does differently from BPE: operates on raw text including whitespace
SentencePiece tokenizes text without pre-tokenization, preserving whitespace
subword tokenization solves: handles rare words by breaking into known pieces
Subword tokenization solves rare word handling by breaking into known pieces
[CLS] pooling does: uses the first token's embedding as the sentence representation
CLS pooling: uses the first token's embedding as the sentence representation
Unigram tokenization does: starts with large vocabulary and prunes using EM
Unigram tokenization starts with a large vocabulary and prunes using EM
Swipe through 100 ML concepts daily
Open Pocket Polymath