the vocabulary size matters: larger vocab = shorter sequences but more parameters

Bigger vocab means shorter sentences but more brainpower needed

Imagine you're trying to explain a complex idea to a friend. You want to be clear but don't want to ramble on forever.

The more words you know, the fewer sentences you need to say something complex. But knowing more words means you have to remember more of them.

Example

Instead of saying "The cat sat on the mat," you could say "The feline rested atop the rug," which is shorter but uses more words.

Remember this

A larger vocabulary allows you to express ideas more concisely, but it also means you need to manage more words in your memory.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

WordPiece tokenization does: similar to BPE but uses likelihood instead of frequency

WordPiece tokenization splits words into subwords based on token likelihood rather than frequency

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

Philosophy of language

Philosophy of language studies language's nature and its relationship with users and the world

word error rate (WER) measures: edit distance between predicted and reference transcriptions

Ever wondered how machines understand speech as we do?

384-dim all-MiniLM-L6-v2 optimizes: fast sentence similarity with 6 layers

All-MiniLM-L6-v2 optimizes fast sentence similarity with 6 layers

ring attention does: distributes long sequences across multiple devices

How can a machine understand and generate human language?

Swipe through 100 ML concepts daily