BPE tokenization does: iteratively merges the most frequent byte pairs

How can we shrink text files without losing meaning?

Imagine you have a huge text file filled with common phrases and words. You want to compress it to save space on your computer, but you don't want to lose any important information.

The idea is to repeatedly find and replace the most common pairs of characters in the text with a single new character, gradually reducing the file size while maintaining the original message.

Example

In a text with the phrase "to be or not to be," the most common pair is "to," which is replaced with a new character, like "T."

Remember this

BPE tokenization compresses text by replacing frequent character pairs with single tokens.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

BPE tokenization does: iteratively merges the most frequent adjacent byte pairs

How do we make computers understand language better?

WordPiece tokenization does: similar to BPE but uses likelihood instead of frequency

WordPiece tokenization splits words into subwords based on token likelihood rather than frequency

Shannon's source coding theorem: you can't compress below entropy

Can you squeeze endless text into fewer bits without losing anything?

SentencePiece does differently from BPE: operates on raw text including whitespace

SentencePiece tokenizes text without pre-tokenization, preserving whitespace

subword tokenization solves: handles rare words by breaking into known pieces

Subword tokenization solves rare word handling by breaking into known pieces

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

Swipe through 100 ML concepts daily

Open Pocket Polymath