Softmax function

Softmax converts real numbers into a probability distribution

The softmax function takes a vector of real numbers and applies the exponential function to each element, then normalizes these values by dividing by the sum of all exponentials. This transformation ensures that the output values are non-negative and sum up to one, making them a valid probability distribution. The softmax function is particularly useful in neural networks for tasks like classification, where it helps to convert the raw output scores into probabilities for each class.

Example

Given a vector [2, 1, 0.1], the softmax function will first compute the exponentials: exp(2) = 7.389, exp(1) = 2.718, exp(0.1) = 1.105. Then, it normalizes these values by dividing each by the sum of all exponentials: 7.389 + 2.718 + 1.105 = 11.212. The resulting softmax probabilities are approximately [0.655, 0.245, 0.010].

Remember this

Understanding the softmax function is crucial for interpreting neural network outputs in classification tasks, as it provides a clear probability distribution over possible outcomes.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

to write a fused softmax kernel in Triton: load row, compute max, subtract, exp, sum, divide

What if we could turn raw scores into probabilities?

Write the attention score formula before softmax: e_ij = a(s_i, h_j)

How do we know what's important in a sentence?

Normal distribution

Normal distribution PDF formula

Akaike information criterion

How do we choose the best model for our data?

Normalization (machine learning)

L2 normalization equation: x_i' = x_i / ||x||_2

Exponential distribution

Why do we sometimes need to predict rare events?

Swipe through more Machine Learning concepts

Open Pocket Polymath