Adam combines momentum and RMSprop: adapts per-parameter learning rates

Ever wondered how a car adjusts its speed for smooth turns?

Imagine you're driving a car on a winding road; you need to adjust your speed smoothly to avoid skidding or crashing.

Adam's method is like adjusting your car's speed for each turn, ensuring a smooth ride. It combines the car's momentum with an average of past speeds, adapting the speed for each turn.

Example

If you're turning left and your car's speed is 60 mph, Adam's method might reduce it to 45 mph for a smoother turn.

Remember this

Adam's method adapts your car's speed for each turn, ensuring a smooth and safe ride.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

AdaGrad's learning rate decays to zero

Why does a car's speed drop when it goes uphill?

Adam vs SGD: Adam adapts per-parameter rates, SGD often generalizes better with tuning

Adam adjusts learning rates per-parameter, SGD generalizes better with tuning

MoE models have more parameters but similar compute cost

MoE models distribute parameters across k experts, reducing active experts' compute cost

autoencoders learn the data manifold

Autoencoders compress data manifold by forcing information through a bottleneck layer, learning efficient representations

instruction tuning does: fine-tunes on (instruction, response) pairs

Fine-tuning adapts pre-trained models to new tasks

LAMB optimizer does: layer-wise adaptive learning rates for large batch training

LAMB optimizer adjusts learning rates layer-wise for large batch training

Swipe through 100 ML concepts daily

Open Pocket Polymath