expected calibration error (ECE) measures: gap between confidence and accuracy

calibration means: a model predicting 80% should be correct 80% of the time

A model predicting 80% should be correct 80% of the time

Brier score

Brier score measures mean squared error of probability predictions

word error rate (WER) measures: edit distance between predicted and reference transcriptions

WER measures the percentage of errors in transcription

Binomial proportion confidence interval

Binomial proportion confidence interval estimates success probability

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

Cross-entropy loss penalizes confident wrong predictions more heavily

to normalize features: when features have different scales and you use distance-based methods

Why do some things need to be adjusted to compare fairly?