the compute-optimal training ratio is: roughly 20 tokens per parameter

How can we train AI efficiently without wasting resources?

Imagine you're teaching a dog new tricks. You want to avoid overwhelming it with too many commands at once, right?

Think about it like this: if you give your dog too many treats at once, it won't learn the trick well. You need to find the right balance. The technical term for this balance is the "compute-optimal training ratio."

Example

If you give your dog 20 treats (tokens) for every trick (parameter) it learns, it can learn efficiently without getting confused.

Remember this

The key insight is to use around 20 tokens per parameter to train AI effectively.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

MoE models have more parameters but similar compute cost

MoE models distribute parameters across k experts, reducing active experts' compute cost

gradient accumulation simulates larger batch sizes without more memory

Can you train a machine like you do with a computer?

the embedding layer does: maps discrete token IDs to dense learned vectors

Embeddings convert token IDs to dense vectors for neural network processing

gradient checkpointing trades: recomputes activations to save memory

Gradient checkpointing trades off computation time for memory savings by recomputing activations

LAMB optimizer does: layer-wise adaptive learning rates for large batch training

LAMB optimizer adjusts learning rates layer-wise for large batch training

quantization to INT8 doubles throughput

Quantization to INT8 doubles throughput because tensor cores process INT8 2x faster

Swipe through 100 ML concepts daily

Open Pocket Polymath