learning rate warmup does: starts small to avoid early training instability

Why does starting big in learning sometimes lead to chaos?

Imagine you're baking a cake for the first time. You mix up too much flour and sugar at once, and the batter becomes lumpy and unstable.

In learning, jumping straight into complex tasks can be overwhelming, like mixing too much flour and sugar at once. Starting small helps you get the hang of things without getting overwhelmed.

Example

If you start with 10 cups of flour and 10 cups of sugar all at once, the batter becomes lumpy. But if you start with 1 cup of flour and 1 cup of sugar, you can mix it perfectly.

Remember this

Starting small helps avoid early training instability. (Learning rate warmup)

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

AdaGrad's learning rate decays to zero

Why does a car's speed drop when it goes uphill?

weight initialization matters: Xavier/He init keeps activation variance ≈ 1 across layers

Why does starting a video game with random settings affect gameplay?

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

LAMB optimizer does: layer-wise adaptive learning rates for large batch training

LAMB optimizer adjusts learning rates layer-wise for large batch training

ill-conditioned matrices cause numerical instability: small input changes → large output changes

Ill-conditioned matrices lead to numerical instability

gradient accumulation simulates larger batch sizes without more memory

Can you train a machine like you do with a computer?

Swipe through 100 ML concepts daily

Open Pocket Polymath