How do we avoid overshooting in learning?
Image: National Oceanic and Atmospheric Administration, Public domain, via Wikimedia Commons
AdaGrad does: divides learning rate by sqrt of sum of squared gradients
How do we avoid overshooting in learning?
Imagine you're learning to ride a bike. If you push too hard, you might fall over. You need to find the right balance in your movements.
Think of adjusting your speed as you ride. You don't want to go too fast and lose control, but you also don't want to go too slow to get nowhere. AdaGrad helps by adjusting your speed based on how much you're pushing and falling over.
Example
If you push too hard (large gradients) and fall (large loss), AdaGrad reduces your speed (learning rate) more for those big pushes.
Remember this
AdaGrad adjusts your learning speed based on past mistakes, helping you find the right balance without falling over.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
AdaGrad's learning rate decays to zero
Why does a car's speed drop when it goes uphill?
Neural network (machine learning)
Ever tried adjusting the learning rate like tuning a musical instrument?
gradient accumulation simulates larger batch sizes without more memory
Can you train a machine like you do with a computer?
Proximal gradient methods for learning
Why can't we always find the best path in a maze?
score matching does: learns the gradient of the log-density without normalizing
Ever wonder how we can compare apples and oranges fairly in studies?
LAMB optimizer does: layer-wise adaptive learning rates for large batch training
LAMB optimizer adjusts learning rates layer-wise for large batch training
Swipe through 100 ML concepts daily
Open Pocket Polymath