saddle points are more common than local minima in high dimensions

Why do some mountains have flat tops instead of peaks or valleys?

Imagine hiking and noticing that some mountain passes don't lead to higher or lower ground but instead level off at a plateau.

Think of a mountain pass as a path on a curved surface. Saddle points are like flat spots on this surface where the slope doesn't go up or down, but stays level. These are called saddle points.

Example

Picture a mountain pass that starts steeply uphill, then curves and flattens out before descending again. The flat part is the saddle point.

Remember this

Saddle points are more common than peaks or valleys in high-dimensional landscapes because they represent flat spots where the slope is zero in multiple directions.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

non-convex loss landscapes are hard: many local minima and saddle points

Why do some hills have more tricky paths than others?

SGD with momentum escapes local minima better than vanilla SGD

Ever felt stuck on a hill, unable to find the way down?

List of unsolved problems in mathematics

Why do random points in high dimensions seem to be evenly spaced?

Gradient

Gradient points uphill in the direction of steepest increase of f

random projection to O(log n/ε²) dimensions preserves pairwise distances within 1±ε

Can you shrink a 3D object into 2D without losing its shape?

Geodesics on an ellipsoid

Geodesics are the shortest paths on a curved surface

Swipe through 100 ML concepts daily

Open Pocket Polymath