gradient checkpointing trades: recomputes activations to save memory

gradient accumulation simulates larger batch sizes without more memory

Can you train a machine like you do with a computer?

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

Vanishing gradient problem

Residual connections help by allowing gradient flow through the skip connection

Flashbulb memory

Flashbulb memories are vivid but not always accurate

Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates