How do we avoid overshooting in learning?
Image: National Oceanic and Atmospheric Administration, Public domain, via Wikimedia Commons
AdaGrad does: divides learning rate by sqrt of sum of squared gradients
How do we avoid overshooting in learning?
Imagine you're learning to ride a bike. If you push too hard, you might fall over. You need to find the right balance in your movements.
Think of adjusting your speed as you ride. You don't want to go too fast and lose control, but you also don't want to go too slow to get nowhere. AdaGrad helps by adjusting your speed based on how much you're pushing and falling over.
Example
If you push too hard (large gradients) and fall (large loss), AdaGrad reduces your speed (learning rate) more for those big pushes.
Remember this
AdaGrad adjusts your learning speed based on past mistakes, helping you find the right balance without falling over.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
AdaGrad's learning rate decays to zero
Why does a car's speed drop when it goes uphill?
gradient accumulation simulates larger batch sizes without more memory
Can you train a machine like you do with a computer?
Proximal gradient methods for learning
Why can't we always find the best path in a maze?
LAMB optimizer does: layer-wise adaptive learning rates for large batch training
LAMB optimizer adjusts learning rates layer-wise for large batch training
to standardize: when you need zero mean and unit variance for gradient-based optimization
Why do we need to make data uniform before training a model?
batch size affects generalization: larger batches find sharper minima
Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates
Swipe through 100 ML concepts daily
Open Pocket Polymath