calibration means: a model predicting 80% should be correct 80% of the time

A model predicting 80% should be correct 80% of the time

The concept of calibration in predictive models refers to the accuracy of the predicted probabilities. A well-calibrated model will have predictions that match the observed outcomes over time.

Example

If a model predicts that an event has an 80% chance of occurring, then in 80 out of 100 cases, the event should actually happen.

Remember this

Calibration ensures that the confidence of a model's predictions is reliable, which is crucial for making informed decisions based on those predictions.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

expected calibration error (ECE) measures: gap between confidence and accuracy

Expected Calibration Error (ECE) measures the gap between predicted confidence levels and actual accuracy

word error rate (WER) measures: edit distance between predicted and reference transcriptions

WER measures the percentage of errors in transcription

importance sampling does: reweights samples from proposal to estimate target expectation

Importance sampling estimates target expectations using samples from a different distribution

to standardize: when you need zero mean and unit variance for gradient-based optimization

Why do we need to make data uniform before training a model?

P-value

A p-value < 0.05 means: if H₀ is true, this result has <5% probability

Filtration (mathematics)

How can we predict future events with uncertainty?

Swipe through 100 ML concepts daily

Open Pocket Polymath