How do we measure the unpredictability of an event?
Image: Unknown authorUnknown author, CC BY 4.0, via Wikimedia Commons
Rényi entropy
How do we measure the unpredictability of an event?
Imagine you're flipping a coin. You know it can land heads or tails, but you can't predict which one it will be. This unpredictability is something Shannon entropy measures.
Shannon entropy quantifies the uncertainty or surprise in the outcome of flipping a coin. It tells us how much information we gain when we find out the result.
Example
Flipping a coin once gives us 1 bit of Shannon entropy because we don't know if it's heads or tails.
Remember this
Shannon entropy measures the unpredictability of an event, telling us how much information we gain when we find out the result.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits
How do we measure uncertainty in everyday decisions?
Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p
Ever wondered how well we can guess the outcome of a random event?
Entropy in thermodynamics and information theory
Ever wondered how computers decide what's important in a message?
Information theory
How much do you know before flipping a coin?
temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform
How does adjusting T affect the certainty of choices?
Likelihood function
Why can't we just guess probabilities correctly all the time?
Swipe through more Machine Learning concepts
Open Pocket Polymath