AdaGrad's learning rate decays to zero

Why does a car's speed drop when it goes uphill?

Imagine you're driving a car uphill; it feels like the engine struggles more as you ascend, making it harder to maintain speed.

Think of AdaGrad adjusting the car's speed uphill. It starts fast but slows down as it goes higher, trying to conserve energy for the steep climb.

Example

If you push the gas pedal hard at first, the car speeds up quickly, but as you keep pushing, it feels like the gas pedal becomes harder to press, slowing you down.

Remember this

AdaGrad's learning rate decreases over time because the denominator in its formula grows, making it harder to increase the learning rate.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Adam combines momentum and RMSprop: adapts per-parameter learning rates

Ever wondered how a car adjusts its speed for smooth turns?

AdaGrad does: divides learning rate by sqrt of sum of squared gradients

How do we avoid overshooting in learning?

ill-conditioned matrices cause numerical instability: small input changes → large output changes

Ill-conditioned matrices lead to numerical instability

gradient checkpointing trades: recomputes activations to save memory

Gradient checkpointing trades off computation time for memory savings by recomputing activations

gradient accumulation simulates larger batch sizes without more memory

Can you train a machine like you do with a computer?

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

Swipe through 100 ML concepts daily

Open Pocket Polymath