non-convex loss landscapes are hard: many local minima and saddle points

Why do some hills have more tricky paths than others?

Imagine hiking in a mountainous region with many peaks and valleys. Some paths seem to lead to higher ground, but others loop back down or lead to flat areas.

The landscape of the hills is like a complex puzzle with many paths. Some paths lead to higher ground (local maxima), others loop back down (saddle points), and some even lead to flat areas (local minima). The challenge is finding the best path to the highest peak (global maximum).

Example

You hike up a steep path (convex function), but it leads to a flat plateau (local minimum). You then take a winding path (non-convex function) that seems to have many ups and downs, making it harder to find the highest peak (global maximum).

Remember this

Non-convex landscapes have many paths that seem promising but ultimately don't lead to the best outcome (global maximum).

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

saddle points are more common than local minima in high dimensions

Why do some mountains have flat tops instead of peaks or valleys?

SGD with momentum escapes local minima better than vanilla SGD

Ever felt stuck on a hill, unable to find the way down?

t-SNE preserves local structure

Can we see the hidden patterns in a cloud of data points?

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

Ridge regression uses L2 to shrink coefficients without eliminating them

Why do some roads get smoother and straighter over time?

Convex optimization

Convex functions have only one global minimum

Swipe through 100 ML concepts daily

Open Pocket Polymath