log-probabilities are used instead of probabilities: avoids numerical underflow

Why can't we just add up tiny chances over time?

Imagine you're rolling a fair six-sided die repeatedly. Each roll is independent, and you're curious about the chance of rolling a six after many rolls.

As you keep rolling, the chance of rolling a six each time stays the same. But when you try to add up all those tiny chances over time, they can become incredibly small numbers that computers struggle to handle.

Example

After 100 rolls, the chance of rolling a six each time is 1/6. Adding up 100 tiny chances (1/6 each) gives you a tiny number that's hard to work with.

Remember this

Log-probabilities help us avoid tiny numbers that computers can't handle well, making calculations easier.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

cross-entropy equals negative log-likelihood for classification

Why does knowing the wrong probability help us measure information loss?

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

to use log-transform: when data is right-skewed or spans multiple orders of magnitude

Why can't we just add numbers to compare incomes?

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

Cross-entropy loss penalizes confident wrong predictions more heavily

the Dirichlet distribution does: distribution over probability simplices

How do we predict the likelihood of various outcomes in uncertain situations?

Bias vs variance: high bias = underfitting, high variance = overfitting

Can a perfect fit to past data predict future events?

Swipe through 100 ML concepts daily

Open Pocket Polymath