BPE tokenization does: iteratively merges the most frequent adjacent byte pairs

How do we make computers understand language better?

Imagine you want to send a secret message to a friend using only a limited set of symbols. You can only use pairs of symbols that appear next to each other in common phrases.

By looking at the most common pairs of symbols in phrases, we can start combining them to create longer, meaningful sequences. This process gradually builds up a dictionary of symbol pairs.

Example

If "hi" and "bye" are common, we start with "hi" and then add "bye" to get "hiby."

Remember this

This method, called BPE (Byte Pair Encoding), helps computers learn language by focusing on the most frequent symbol pairs.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

BPE tokenization does: iteratively merges the most frequent byte pairs

How can we shrink text files without losing meaning?

WordPiece tokenization does: similar to BPE but uses likelihood instead of frequency

WordPiece tokenization splits words into subwords based on token likelihood rather than frequency

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

Large language model

LLMs can generate, summarize, translate, and analyze text in many contexts

subword tokenization solves: handles rare words by breaking into known pieces

Subword tokenization solves rare word handling by breaking into known pieces

Overlapping subproblems

Ever calculated a huge Fibonacci sequence by hand?

Swipe through 100 ML concepts daily

Open Pocket Polymath