calibration means: a model predicting 80% should be correct 80% of the time

How can we trust a coin flip to predict the weather?

Imagine you're trying to guess if it will rain tomorrow. You flip a coin and say heads means rain and tails means no rain. You want to be right 80% of the time.

Flipping a coin isn't a good way to predict rain because it's random. Instead, we need a method that consistently gets the right answer most of the time.

Example

You flip the coin 10 times and get heads 8 times, but it actually rained 6 times. Your coin flip isn't a reliable predictor.

Remember this

Calibration means having a method that gives the correct answer 80% of the time, not relying on random guesses.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Write the reparameterization trick z = μ + σ⊙ε

How can you predict the weather without getting wet?

expected calibration error (ECE) measures: gap between confidence and accuracy

Expected Calibration Error (ECE) measures the gap between predicted confidence levels and actual accuracy

Finite element method

Why does a straight-line guess fail in complex terrain?

importance sampling does: reweights samples from proposal to estimate target expectation

Why can't we always use the same samples to figure out what's happening in a complex system?

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

How wrong predictions hurt more than they're supposed to

temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform

How does adjusting T affect the certainty of choices?

Swipe through 100 ML concepts daily

Open Pocket Polymath