Greedy vs beam search decoding: greedy picks best token, beam maintains k candidates

Ever wondered why Google Search sometimes shows you the top results first?

Imagine you're searching for recipes online and want to find the best one quickly without scrolling endlessly.

Think of it like picking the tastiest-looking fruit from a bunch to try first, instead of trying every single one. In this case, Google uses a method called "beam search" to quickly find the best results without checking every single option.

Example

If you have 100 recipes and Google only checks the top 10 based on your search, it saves time and energy.

Remember this

Beam search keeps only the top 10 promising results, making it faster than checking every single recipe.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Greedy vs dynamic programming: greedy makes locally optimal choices, DP considers all subproblems

Greedy: locally optimal choices; DP: considers all subproblems

Top-k vs top-p sampling: top-k fixes candidate count, top-p fixes cumulative probability mass

Top-k sampling fixes candidate count; top-p sampling fixes cumulative probability mass

subword tokenization solves: handles rare words by breaking into known pieces

Subword tokenization solves rare word handling by breaking into known pieces

Mixture of experts

Mixture of experts (MoE) divides problem space into homogeneous regions

Encoder vs decoder: encoder sees all tokens bidirectionally, decoder sees only past tokens

Encoder: Sees all tokens bidirectionally; Decoder: Sees only past tokens

GQA reduces KV-cache memory by the group factor

Ever wondered how websites stay fresh in search results?

Swipe through 100 ML concepts daily

Open Pocket Polymath