Why can't we just use one score for everything?
Evaluation of binary classifiers
Why can't we just use one score for everything?
Imagine you're comparing two weather apps to predict rain. One app is great at telling you when it's going to rain (high sensitivity), but misses a lot of sunny days (low specificity). The other app is great at telling you when it's not going to rain (high specificity), but misses some rainy days (low sensitivity).
If you want to know how reliable an app is overall, you can't just look at sunny days or rainy days separately. You need a way to see how well it predicts both, without caring if it's sunny or rainy.
Example
App A says it will rain 90% of the time it actually does, but it only says sunny 40% of the time when it's really sunny. App B says it won't rain 90% of the time it actually doesn't, but it only says sunny 60% of the time when it's really sunny.
Remember this
The Matthews correlation coefficient (MCC) is preferred when you need a single score that considers both the sensitivity and specificity of a binary classification model, regardless of the prevalence of sunny or rainy days.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
F-score
Why is one metric not enough to judge a binary classification model's performance?
Phi coefficient
Matthews correlation coefficient (MCC) measures balanced metric even with class imbalance
to use F1 score: when classes are imbalanced and both FP and FN matter
Why might a sports team need a different score to judge their performance?
Binary classification
Why is it important to know if something is a disease or not?
to use AUC-ROC: comparing classifiers across all thresholds
Why can't we always trust a yes-or-no answer?
score matching does: learns the gradient of the log-density without normalizing
Ever wonder how we can compare apples and oranges fairly in studies?
Swipe through more Machine Learning concepts
Open Pocket Polymath