gradient checkpointing trades: recomputes activations to save memory

gradient accumulation simulates larger batch sizes without more memory

Can you train a machine like you do with a computer?

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

Vanishing gradient problem

Residual connections help by allowing gradient flow through the skip connection

Flashbulb memory

Flashbulb memories are vivid but not always accurate

Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

instruction tuning does: fine-tunes on (instruction, response) pairs

Fine-tuning adapts pre-trained models to new tasks