SGD with momentum escapes local minima better than vanilla SGD

Ever felt stuck on a hill, unable to find the way down?

Imagine you're hiking and suddenly realize you're on a hill instead of a trail. You want to find the lowest point to start your descent safely.

Stochastic Gradient Descent (SGD) with momentum is like taking small, random steps downhill, but with a twist that helps you slide over small bumps more smoothly and consistently, avoiding getting stuck on small hills.

Example

Say you're walking down a hill (the hill represents a high-dimensional optimization problem). Normally, you might take a step forward (vanilla SGD), but with SGD with momentum, you take a step forward and then slide a bit more in the same direction, helping you avoid getting stuck on small hills (local minima).

Remember this

SGD with momentum helps escape local minima by combining random steps with a sliding motion, making it easier to find the lowest point.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

non-convex loss landscapes are hard: many local minima and saddle points

Why do some hills have more tricky paths than others?

saddle points are more common than local minima in high dimensions

Why do some mountains have flat tops instead of peaks or valleys?

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

AdaGrad's learning rate decays to zero

Why does a car's speed drop when it goes uphill?

the momentum term does: v_t = βv_{t-1} + ∇L, accumulates gradient direction

Momentum term accelerates convergence in the gradient direction

Convex optimization

Convex functions have only one global minimum

Swipe through 100 ML concepts daily

Open Pocket Polymath