How does adjusting T affect the certainty of choices?
Image: Trondheim Havn from Trondheim, Norway, CC BY-SA 2.0, via Wikimedia Commons
temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform
How does adjusting T affect the certainty of choices?
Imagine you're choosing a meal from a menu, and you want to be sure about your choice.
If T is very small, you're almost sure to pick the best option, like a top chef's recommendation. But if T is very large, you're just picking randomly, like flipping a coin for each dish.
Example
With T=1, you might have a 70% chance of picking your favorite dish, but with T=100, it's like choosing blindly.
Remember this
Small T makes you confident in your choice, while large T makes it random.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits
How do we measure uncertainty in everyday decisions?
Rényi entropy
How do we measure the unpredictability of an event?
Exponential distribution
Why do we sometimes need to predict rare events?
the condition number κ(A) measures: sensitivity of Ax=b to perturbations
How small changes affect big outcomes
Information theory
How much do you know before flipping a coin?
log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily
How wrong predictions hurt more than they're supposed to
Swipe through more Machine Learning concepts
Open Pocket Polymath