Kullback–Leibler divergence

KL divergence is not symmetric: D_KL(P||Q) ≠ D_KL(Q||P)

The Kullback–Leibler (KL) divergence measures how much one probability distribution diverges from a second, expected probability distribution. It is not symmetric, meaning D_KL(P||Q) is not equal to D_KL(Q||P).

Mathematically, KL divergence is defined as the sum over all possible outcomes x in the set X of P(x) multiplied by the logarithm of P(x) divided by Q(x). This definition highlights that KL divergence quantifies the information lost when Q approximates P.

The lack of symmetry in KL divergence implies that the divergence from P to Q is not the same as the divergence from Q to P. This asymmetry is crucial in applications such as machine learning, where it affects how models are trained and evaluated.

Example

Consider two distributions P and Q where P(x) = 0.5 for x = 1 and P(x) = 0 for x = 2, and Q(x) = 0.25 for x = 1 and Q(x) = 0.75 for x = 2. Calculating D_KL(P||Q) and D_KL(Q||P) will yield different results, demonstrating the asymmetry.

Remember this

Understanding the asymmetry of KL divergence is essential for correctly interpreting and applying it in statistical modeling and machine learning tasks.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

KL divergence is always ≥ 0 and equals 0 only when P = Q exactly

Why can't we just compare two things directly?

Jensen–Shannon divergence

Jensen-Shannon divergence formula: D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M))

Cholesky decomposition

Cholesky decomposition factors A = LL^T for symmetric positive definite matrices

the L1 norm is not differentiable at zero

Absolute value's kink makes it non-differentiable at zero

the trace equals the sum of eigenvalues: tr(A) = Σλ_i

How can a vector stay the same after a transformation?

the momentum term does: v_t = βv_{t-1} + ∇L, accumulates gradient direction

Momentum term accelerates convergence in the gradient direction

Swipe through 100 ML concepts daily

Open Pocket Polymath