Brier score

Brier score measures mean squared error of probability predictions

The Brier score quantifies the accuracy of probabilistic predictions by calculating the mean squared error between predicted probabilities and actual outcomes. It is a strictly proper scoring rule, ensuring that better predictions always receive a lower score. The score is applicable to tasks with mutually exclusive outcomes, where probabilities must sum to one.

Example

If a weather forecast predicts a 70% chance of rain and it rains, the Brier score for this prediction would be calculated based on the squared difference between the predicted probability (0.7) and the actual outcome (1 for rain, 0 for no rain).

Remember this

Understanding the Brier score helps in evaluating and improving the accuracy of probabilistic predictions in various fields.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

the Dirichlet distribution does: distribution over probability simplices

How do we predict the likelihood of various outcomes in uncertain situations?

score matching does: learns the gradient of the log-density without normalizing

Ever wonder how we can compare apples and oranges fairly in studies?

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

BLEU vs ROUGE: BLEU measures precision of n-grams, ROUGE measures recall

BLEU measures precision of n-grams, ROUGE measures recall

denoising score matching does: learns to denoise, which equals learning the score

Can you clean up a noisy picture?

expected calibration error (ECE) measures: gap between confidence and accuracy

Expected Calibration Error (ECE) measures the gap between predicted confidence levels and actual accuracy

Swipe through 100 ML concepts daily

Open Pocket Polymath