Adam vs SGD: Adam adapts per-parameter rates, SGD often generalizes better with tuning

Adam combines momentum and RMSprop: adapts per-parameter learning rates

Ever wondered how a car adjusts its speed for smooth turns?

the β₁ and β₂ hyperparameters control in Adam

β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

instruction tuning does: fine-tunes on (instruction, response) pairs

Fine-tuning adapts pre-trained models to new tasks

MoE models have more parameters but similar compute cost

MoE models distribute parameters across k experts, reducing active experts' compute cost

data augmentation does for generalization: artificially expands training set

How can you teach a computer to see better?