Entropy (information theory)

Entropy of a fair coin is 1 bit

Entropy measures uncertainty or information content. A fair coin has two equally likely outcomes, heads or tails. Entropy quantifies the average amount of information needed to describe the coin's state.

A fair coin's entropy is calculated using the formula H(X) = -∑ p(x) log₂ p(x). With equal probabilities (p(heads) = 0.5, p(tails) = 0.5), the entropy is H(X) = -[0.5 log₂(0.5) + 0.5 log₂(0.5)] = 1 bit.

This concept is fundamental in information theory, helping to understand data compression and transmission efficiency.

Example

For a fair coin, H(X) = 1 bit, indicating maximum uncertainty or information content.

Remember this

Entropy quantifies uncertainty, crucial for data compression and transmission efficiency.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

A fair die has entropy of log₂(6) ≈ 2.58 bits

How much information do you need to guess a die roll?

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform

How does adjusting T affect the certainty of choices?

Shannon's source coding theorem: you can't compress below entropy

Can you squeeze endless text into fewer bits without losing anything?

Perplexity

Perplexity = 2^H

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Ever wondered how well we can guess the outcome of a random event?

Swipe through 100 ML concepts daily

Open Pocket Polymath