Adam vs SGD: Adam adapts per-parameter rates, SGD often generalizes better with tuning

Adam combines momentum and RMSprop: adapts per-parameter learning rates

Ever wondered how a car adjusts its speed for smooth turns?

Neural network (machine learning)

Ever tried adjusting the learning rate like tuning a musical instrument?

the β₁ and β₂ hyperparameters control in Adam

β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

Fine-tuning (deep learning)

Why can we make a big change with just a tiny tweak?

MoE models have more parameters but similar compute cost

MoE models distribute parameters across k experts, reducing active experts' compute cost