Bias vs variance: high bias = underfitting, high variance = overfitting

Can a perfect fit to past data predict future events?

Imagine you're trying to predict the weather for your picnic next week. You've noticed that when you used a simple weather app, it often got it wrong, especially for unusual weather patterns.

You need a balance. A simple app might not catch all the details (high bias), while a super complex app might just guess wildly (high variance). The key is finding the sweet spot.

Example

Your simple app predicted sunny (high bias), but the complex app predicted rain (high variance) for your picnic.

Remember this

The sweet spot is where your predictions are accurate without being too simple or too complex.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

Cross-entropy loss penalizes confident wrong predictions more heavily

Boosting (machine learning)

Boosting reduces bias in ML models

Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

to standardize: when you need zero mean and unit variance for gradient-based optimization

Why do we need to make data uniform before training a model?

log-probabilities are used instead of probabilities: avoids numerical underflow

Why can't we just add up tiny chances over time?

to use random forests: when you want a strong baseline with minimal hyperparameter tuning

Why not always use the best single tree for predictions?

Swipe through 100 ML concepts daily

Open Pocket Polymath