A fair die has entropy of log₂(6) ≈ 2.58 bits

How much information do you need to guess a die roll?

Imagine you're playing a board game with a six-sided die. You want to know how many bits of information you'd need to describe the outcome of each roll.

Think of each possible outcome as a unique story. The more stories there are, the more bits you need to tell them. For a die, there are six outcomes, so you need log₂(6) bits to describe any roll.

Example

If you had 2 outcomes (like flipping a coin), you'd need log₂(2) = 1 bit. But with 6 outcomes (like rolling a die), you need log₂(6) ≈ 2.58 bits.

Remember this

Entropy measures the average level of uncertainty or information needed to describe the outcomes of a random variable.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

Entropy (information theory)

Entropy of a fair coin is 1 bit

cross-entropy equals negative log-likelihood for classification

Why does knowing the wrong probability help us measure information loss?

Entropy in thermodynamics and information theory

Ever wondered how computers decide what's important in a message?

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Ever wondered how well we can guess the outcome of a random event?

Shannon's source coding theorem: you can't compress below entropy

Can you squeeze endless text into fewer bits without losing anything?

Swipe through 100 ML concepts daily

Open Pocket Polymath