Entropy in thermodynamics and information theory

Ever wondered how computers decide what's important in a message?

Imagine you're sending a text with lots of emojis, and you want the recipient to know which emoji is most important.

Shannon entropy helps us figure out which emoji is the most crucial by measuring how much surprise or uncertainty there is in the emoji choices.

Example

If you sent 10 texts with different emoji combinations, Shannon entropy would tell you which combination was the most unpredictable and thus, potentially the most important.

Remember this

Shannon entropy quantifies the unpredictability of information, guiding us on what to expect next.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Ever wondered how well we can guess the outcome of a random event?

A fair die has entropy of log₂(6) ≈ 2.58 bits

How much information do you need to guess a die roll?

Shannon's source coding theorem: you can't compress below entropy

Can you squeeze endless text into fewer bits without losing anything?

temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform

How does adjusting T affect the certainty of choices?

Mutual information

Mutual information formula: I(X;Y) = ∑_x∈X ∑_y∈Y p(x,y) log(p(x,y)/(p(x)p(y)))

Swipe through 100 ML concepts daily

Open Pocket Polymath