How do we measure uncertainty in everyday decisions?
Image: Leenaborbarua, CC BY-SA 4.0, via Wikimedia Commons
Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits
How do we measure uncertainty in everyday decisions?
Imagine you're flipping a coin, unsure if it lands heads or tails. You want to predict the outcome but don't know the probability.
Entropy helps us understand the unpredictability of outcomes like coin flips. It quantifies the average surprise or information needed to know the result.
Example
Flipping a fair coin 100 times, you expect about 50 heads and 50 tails. Entropy tells us the average surprise in predicting each flip's outcome.
Remember this
Entropy measures the average surprise or uncertainty in outcomes, helping us predict and understand random events.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p
Ever wondered how well we can guess the outcome of a random event?
cross-entropy equals negative log-likelihood for classification
Why does knowing the wrong probability help us measure information loss?
A fair die has entropy of log₂(6) ≈ 2.58 bits
How much information do you need to guess a die roll?
temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform
How does adjusting T affect the certainty of choices?
Entropy in thermodynamics and information theory
Ever wondered how computers decide what's important in a message?
Fisher information
Fisher information measures information about unknown parameters
Swipe through 100 ML concepts daily
Open Pocket Polymath