Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Ever wondered how well we can guess the outcome of a random event?

Imagine you're trying to guess the color of a marble drawn from a bag. If you knew the bag had 50% red and 50% blue marbles, you'd expect to guess correctly 50% of the time. But what if you guessed based on your own guessings, like thinking red is more likely because you've drawn more red marbles so far?

The cross-entropy concept helps us understand how well our guesses (q) match the actual probabilities (p) of drawing a marble. It's like measuring how surprised we are when our guesses don't match reality.

Example

If you guessed red 70% of the time but red was only 50% likely, you'd be surprised 70% of the time when you guess red.

Remember this

Cross-entropy quantifies the surprise or mismatch between our estimated probabilities and the actual probabilities.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Cross-entropy

Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))

Rényi entropy

How do we measure the unpredictability of an event?

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

Information theory

How much do you know before flipping a coin?

Likelihood function

Why can't we just guess probabilities correctly all the time?

cross-entropy equals negative log-likelihood for classification

Why does knowing the wrong probability help us measure information loss?

Swipe through more Machine Learning concepts

Open Pocket Polymath