Jensen-Shannon divergence formula: D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M))
Image: Edward Larsson, Public domain, via Wikimedia Commons
Jensen–Shannon divergence
Jensen-Shannon divergence formula: D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M))
The Jensen-Shannon divergence formula is a symmetric measure of similarity between two probability distributions P and Q. It is derived from the Kullback-Leibler divergence by averaging the two distributions (M = 1/2 * (P + Q)) and then calculating the divergence from each distribution to the average.
The formula D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M)) ensures that the divergence is always finite and symmetric, unlike the Kullback-Leibler divergence. This makes it a more robust measure for comparing probability distributions.
The Jensen-Shannon distance, which is the square root of the Jensen-Shannon divergence, provides a metric for quantifying the similarity between distributions. A smaller Jensen-Shannon distance indicates greater similarity between the distributions.
Example
Consider two probability distributions P and Q, where P = [0.1, 0.9] and Q = [0.8, 0.2]. The average distribution M = [0.45, 0.65]. The Jensen-Shannon divergence D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M)) can be calculated using the Kullback-Leibler divergence formula.
Remember this
Understanding the Jensen-Shannon divergence formula is crucial for accurately measuring the similarity between probability distributions in various fields such as machine learning, information theory, and statistics.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Kullback–Leibler divergence
KL divergence is not symmetric: D_KL(P||Q) ≠ D_KL(Q||P)
Cross-entropy
Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))
KL divergence is always ≥ 0 and equals 0 only when P = Q exactly
Why can't we just compare two things directly?
Discrete Fourier transform
Discrete Fourier Transform (DFT) equation: X[k] = Σ(n=0 to N-1) x[n] * e^(-j*2π*k*n/N)
Expected value
Expected value formula: E[X] = Σ [x * P(x)]
Perplexity
Perplexity = 2^H
Swipe through 100 ML concepts daily
Open Pocket Polymath