Information theory

How much do you know before flipping a coin?

Imagine you're guessing if a coin will land heads or tails. Before flipping, your knowledge is uncertain.

Your uncertainty before flipping is like not knowing anything. The formula H(X) = -Σ P(x) log₂ P(x) measures this uncertainty.

Example

With a fair coin, the probability of heads or tails is 1/2. The uncertainty (information) is -log₂(1/2) = 1 bit.

Remember this

The formula quantifies how much uncertainty (information) you have before knowing the coin's outcome.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

Entropy (information theory)

Entropy of a fair coin is 1 bit

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Ever wondered how well we can guess the outcome of a random event?

A fair die has entropy of log₂(6) ≈ 2.58 bits

How much information do you need to guess a die roll?

Cross-entropy

Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))

Entropy in thermodynamics and information theory

Ever wondered how computers decide what's important in a message?

Swipe through 100 ML concepts daily