Perplexity

Perplexity = 2^H

Perplexity measures uncertainty in a probability distribution. It is calculated as 2 raised to the power of the information entropy (H). Perplexity quantifies how well a probability distribution predicts outcomes.

Example

For a fair coin (N=2 outcomes), H = log2(1/0.5) = 1. Perplexity = 2^1 = 2.

Remember this

Understanding perplexity helps in evaluating models' performance in predicting outcomes, crucial for applications like speech recognition.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Cross-entropy

Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Ever wondered how well we can guess the outcome of a random event?

Jensen–Shannon divergence

Jensen-Shannon divergence formula: D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M))

Entropy (information theory)

Entropy of a fair coin is 1 bit

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

Mutual information

Mutual information formula: I(X;Y) = ∑_x∈X ∑_y∈Y p(x,y) log(p(x,y)/(p(x)p(y)))

Swipe through 100 ML concepts daily

Open Pocket Polymath