How wrong predictions hurt more than they're supposed to
Image: U.S. Navy photo by Photographer's Mate 2nd Class Philip A. McDaniel, Public domain, via Wikimedia Commons
log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily
How wrong predictions hurt more than they're supposed to
Imagine you're guessing answers on a multiple-choice quiz. You confidently choose B, but it's actually A. How unfair is that?
Cross-entropy loss punishes confident wrong predictions more than uncertain ones, because it measures how much extra information is needed when your guesses are off-target.
Example
If you guessed B 10 times and A 2 times, cross-entropy loss would penalize you more for those confident B guesses than for the uncertain A guesses.
Remember this
Cross-entropy loss is harsher on confident wrong predictions.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
cross-entropy equals negative log-likelihood for classification
Why does knowing the wrong probability help us measure information loss?
Bias vs variance: high bias = underfitting, high variance = overfitting
Can a perfect fit to past data predict future events?
Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits
How do we measure uncertainty in everyday decisions?
the condition number κ(A) measures: sensitivity of Ax=b to perturbations
How small changes affect big outcomes
Adam has bias correction: divides by (1-β^t) in early steps
Why do we sometimes need to fix mistakes in computer decisions?
log-probabilities are used instead of probabilities: avoids numerical underflow
Why can't we just add up tiny chances over time?
Swipe through 100 ML concepts daily
Open Pocket Polymath