KL divergence is always ≥ 0 and equals 0 only when P = Q exactly

Why can't we just compare two things directly?

Imagine you're trying to match your taste in music with a friend's. You both have playlists, but they're not identical. You want to know how similar they are.

Think of it like trying to find the difference in taste between two fruit juices. The more you taste, the more you'll notice what makes them unique. The Kullback–Leibler divergence is like a taste test that tells you how much one juice differs from the other.

Example

If one juice has 10% apple flavor and the other has 15%, the difference is noticeable. If both juices have 10% apple flavor, there's no difference in taste.

Remember this

The Kullback–Leibler divergence measures how much one probability distribution diverges from another, showing the difference between them.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Kullback–Leibler divergence

KL divergence is not symmetric: D_KL(P||Q) ≠ D_KL(Q||P)

Convergence of random variables

Ever wonder why flipping a coin repeatedly doesn't always predict the next flip?

Jensen–Shannon divergence

Jensen-Shannon divergence formula: D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M))

Hidden Markov model

Ever wondered how you can predict outcomes without seeing all the details?

Markov chain

Why does flipping a coin repeatedly get predictable?

the L1 norm is not differentiable at zero

Absolute value's kink makes it non-differentiable at zero

Swipe through more Machine Learning concepts

Open Pocket Polymath