natural gradient descent does: preconditions with inverse Fisher matrix

Ever wondered how to climb a steep mountain without measuring every step?

Imagine you're hiking up a steep mountain and need to find the best path quickly without checking the entire trail.

Think of it like using a compass to find the steepest part of the mountain instead of feeling the ground under your feet everywhere. This compass guides you to make small, smarter steps up the mountain.

Example

Instead of feeling the ground every meter, you check your compass every 10 meters to decide which way is steeper.

Remember this

Using a compass (stochastic gradient descent) helps you find the steepest path up the mountain more efficiently than checking every step (full gradient descent).

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

Definite matrix

How do we measure the steepness of a hill in all directions?

the reverse process learns: p_θ(x_{t-1}|x_t)

Can we trace back the roots of life?

Fine-tuning (deep learning)

Why can we make a big change with just a tiny tweak?

score matching does: learns the gradient of the log-density without normalizing

Ever wonder how we can compare apples and oranges fairly in studies?

non-convex loss landscapes are hard: many local minima and saddle points

Why do some hills have more tricky paths than others?

Swipe through more Machine Learning concepts

Open Pocket Polymath