Ridge regression uses L2 to shrink coefficients without eliminating them

Why do some roads get smoother and straighter over time?

Imagine you're driving a car on a winding mountain road. Over time, the road gets smoother and straighter as people drive over it, making it easier to drive.

Just like the road gets smoother, ridge regression smooths out the bumps in a road map filled with lots of twists and turns (highly correlated variables). It does this by adding a tiny bit of extra weight to the turns (L2 regularization), making the overall path smoother and easier to follow.

Example

If the road twists 10 times in one mile, ridge regression might add a small weight to each twist, making it 9.8 times instead of 10, smoothing out the path.

Remember this

Ridge regression adds a tiny bit of extra weight (L2 regularization) to smooth out the path (reduce the impact of highly correlated variables) without removing any twists (coefficients).

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

non-convex loss landscapes are hard: many local minima and saddle points

Why do some hills have more tricky paths than others?

AdaGrad's learning rate decays to zero

Why does a car's speed drop when it goes uphill?

LASSO uses L1 to do feature selection by driving coefficients to exactly zero

Why do some numbers disappear when solving complex problems?

to standardize: when you need zero mean and unit variance for gradient-based optimization

Why do we need to make data uniform before training a model?

Regularization (mathematics)

L1 regularization results in sparse solutions

LoRA (machine learning)

LoRA uses r << d for efficient adaptation

Swipe through 100 ML concepts daily

Open Pocket Polymath