log-probabilities are used instead of probabilities: avoids numerical underflow

Why can't we just add up tiny chances over time?

Imagine you're rolling a fair six-sided die repeatedly. Each roll is independent, and you're curious about the chance of rolling a six after many rolls.

As you keep rolling, the chance of rolling a six each time stays the same. But when you try to add up all those tiny chances over time, they can become incredibly small numbers that computers struggle to handle.

Example

After 100 rolls, the chance of rolling a six each time is 1/6. Adding up 100 tiny chances (1/6 each) gives you a tiny number that's hard to work with.

Remember this

Log-probabilities help us avoid tiny numbers that computers can't handle well, making calculations easier.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Likelihood function

Why can't we just guess probabilities correctly all the time?

Exponential distribution

Why do we sometimes need to predict rare events?

cross-entropy equals negative log-likelihood for classification

Why does knowing the wrong probability help us measure information loss?

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

to use log-transform: when data is right-skewed or spans multiple orders of magnitude

Why can't we just add numbers to compare incomes?

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

How wrong predictions hurt more than they're supposed to

Swipe through more Machine Learning concepts

Open Pocket Polymath