Softmax function

Softmax converts real numbers into a probability distribution

The softmax function takes a vector of real numbers and applies the exponential function to each element, then normalizes these values by dividing by the sum of all exponentials. This transformation ensures that the output values are non-negative and sum up to one, making them a valid probability distribution. The softmax function is particularly useful in neural networks for tasks like classification, where it helps to convert the raw output scores into probabilities for each class.

Example

Given a vector [2, 1, 0.1], the softmax function will first compute the exponentials: exp(2) = 7.389, exp(1) = 2.718, exp(0.1) = 1.105. Then, it normalizes these values by dividing each by the sum of all exponentials: 7.389 + 2.718 + 1.105 = 11.212. The resulting softmax probabilities are approximately [0.655, 0.245, 0.010].

Remember this

Understanding the softmax function is crucial for interpreting neural network outputs in classification tasks, as it provides a clear probability distribution over possible outcomes.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

to write a fused softmax kernel in Triton: load row, compute max, subtract, exp, sum, divide

Softmax function converts real numbers to a probability distribution

Write the attention score formula before softmax: e_ij = a(s_i, h_j)

How do we know what's important in a sentence?

Normal distribution

Normal distribution PDF formula

Normalization (machine learning)

L2 normalization equation: x_i' = x_i / ||x||_2

self-attention: Attention(Q,K,V) = softmax(QK^T/√d_k)V

Ever wondered how computers understand what's important in a sentence?

the Dirichlet distribution does: distribution over probability simplices

How do we predict the likelihood of various outcomes in uncertain situations?

Swipe through 100 ML concepts daily

Open Pocket Polymath