gradient accumulation simulates larger batch sizes without more memory

Can you train a machine like you do with a computer?

Imagine you're trying to teach a dog new tricks without buying expensive treats for every single reward.

Instead of rewarding after every tiny success, you wait until the dog learns a bunch of tricks before giving it a big reward. This way, you don't need to buy as many treats.

Example

You teach your dog to sit, stay, and roll over. After it learns all these tricks, you give it a big treat.

Remember this

By rewarding after several successes, you simulate a bigger reward without needing more treats.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

LAMB optimizer does: layer-wise adaptive learning rates for large batch training

LAMB optimizer adjusts learning rates layer-wise for large batch training

gradient checkpointing trades: recomputes activations to save memory

Gradient checkpointing trades off computation time for memory savings by recomputing activations

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

General-purpose computing on graphics processing units

Did you know your computer can do more than just compute numbers?

DDIM does: deterministic sampling for faster generation with fewer steps

DDIM accelerates image generation by deterministically sampling intermediate steps

Swipe through 100 ML concepts daily

Open Pocket Polymath