log-probabilities are used instead of probabilities: avoids numerical underflow

Why can't we just add up tiny chances over time?

Imagine you're rolling a fair six-sided die repeatedly. Each roll is independent, and you're curious about the chance of rolling a six after many rolls.

As you keep rolling, the chance of rolling a six each time stays the same. But when you try to add up all those tiny chances over time, they can become incredibly small numbers that computers struggle to handle.

Example

After 100 rolls, the chance of rolling a six each time is 1/6. Adding up 100 tiny chances (1/6 each) gives you a tiny number that's hard to work with.

Remember this

Log-probabilities help us avoid tiny numbers that computers can't handle well, making calculations easier.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

cross-entropy equals negative log-likelihood for classification

Why does knowing the wrong probability help us measure information loss?

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

to use log-transform: when data is right-skewed or spans multiple orders of magnitude

Why can't we just add numbers to compare incomes?

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

How wrong predictions hurt more than they're supposed to

the optional stopping theorem says about martingales and stopping times

Ever wondered why betting on a roulette wheel doesn't change your odds over time?

the do-calculus does: computes interventional probabilities from observational data

Can we predict the future from the past?

Swipe through 100 ML concepts daily

Open Pocket Polymath