natural gradient descent does: preconditions with inverse Fisher matrix

Ever wondered how to climb a steep mountain without measuring every step?

Imagine you're hiking up a steep mountain and need to find the best path quickly without checking the entire trail.

Think of it like using a compass to find the steepest part of the mountain instead of feeling the ground under your feet everywhere. This compass guides you to make small, smarter steps up the mountain.

Example

Instead of feeling the ground every meter, you check your compass every 10 meters to decide which way is steeper.

Remember this

Using a compass (stochastic gradient descent) helps you find the steepest path up the mountain more efficiently than checking every step (full gradient descent).

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

non-convex loss landscapes are hard: many local minima and saddle points

Why do some hills have more tricky paths than others?

AdaGrad does: divides learning rate by sqrt of sum of squared gradients

How do we avoid overshooting in learning?

to standardize: when you need zero mean and unit variance for gradient-based optimization

Why do we need to make data uniform before training a model?

Ordinary least squares

OLS minimizes squared differences

approximation algorithms guarantee: solution within factor α of optimal

Can we always find the best solution quickly?

Swipe through 100 ML concepts daily

Open Pocket Polymath