Brier score

Brier score measures mean squared error of probability predictions

The Brier score quantifies the accuracy of probabilistic predictions by calculating the mean squared error between predicted probabilities and actual outcomes. It is a strictly proper scoring rule, ensuring that better predictions always receive a lower score. The score is applicable to tasks with mutually exclusive outcomes, where probabilities must sum to one.

Example

If a weather forecast predicts a 70% chance of rain and it rains, the Brier score for this prediction would be calculated based on the squared difference between the predicted probability (0.7) and the actual outcome (1 for rain, 0 for no rain).

Remember this

Understanding the Brier score helps in evaluating and improving the accuracy of probabilistic predictions in various fields.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

denoising score matching does: learns to denoise, which equals learning the score

Propensity score matching (PSM) reduces bias in treatment effect estimates

the Dirichlet distribution does: distribution over probability simplices

How do we predict the likelihood of various outcomes in uncertain situations?

word error rate (WER) measures: edit distance between predicted and reference transcriptions

WER measures the percentage of errors in transcription

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

BLEU vs ROUGE: BLEU measures precision of n-grams, ROUGE measures recall

BLEU measures precision of n-grams, ROUGE measures recall

score matching does: learns the gradient of the log-density without normalizing

Propensity score matching reduces bias in treatment effect estimates

Swipe through 100 ML concepts daily

Open Pocket Polymath