cross-entropy equals negative log-likelihood for classification

Why does knowing the wrong probability help us measure information loss?

Imagine you're guessing someone's birthday. If you're guessing randomly, you're not using any information. But if you guess based on what birthdays are most common, you're using some information. How do we measure how good your guessing is?

The cross-entropy concept measures how much worse your guessing is when you're wrong. It's like a score for how much information you lost by guessing incorrectly.

Example

If you guessed 365 people would have January 1st as their birthday, but only 1 person actually did, you're losing a lot of information.

Remember this

Cross-entropy equals negative log-likelihood because it quantifies the loss in information when our guesses (probabilities) don't match the true probabilities.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

Cross-entropy loss penalizes confident wrong predictions more heavily

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

Fisher information

Fisher information measures information about unknown parameters

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Ever wondered how well we can guess the outcome of a random event?

Chebyshev's inequality

Chebyshev's inequality limits the probability of deviation from the mean

log-probabilities are used instead of probabilities: avoids numerical underflow

Why can't we just add up tiny chances over time?

Swipe through 100 ML concepts daily

Open Pocket Polymath