Why does starting a video game with random settings affect gameplay?
Image: Billy69150 (voir les conditions d'utilisation / see licensing below), CC BY-SA 4.0, via Wikimedia Commons
weight initialization matters: Xavier/He init keeps activation variance ≈ 1 across layers
Why does starting a video game with random settings affect gameplay?
Imagine you're playing a new video game and it starts with random settings. Some settings make the game too hard or too easy, affecting your enjoyment.
Think of the game's settings as the starting point for the game's action. If the starting point is too extreme, the action becomes too intense or too dull. The goal is to have a balanced starting point so the action stays exciting. This balanced starting point is called Xavier/He initialization.
Example
If the game's difficulty setting starts at 10 (too hard) instead of 5 (balanced), the action becomes too intense, making it hard to enjoy.
Remember this
Xavier/He initialization ensures the game's action stays balanced and enjoyable from the start.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Adam has bias correction: divides by (1-β^t) in early steps
Why do we sometimes need to fix mistakes in computer decisions?
instruction tuning does: fine-tunes on (instruction, response) pairs
Fine-tuning adapts pre-trained models to new tasks
gradient checkpointing trades: recomputes activations to save memory
Gradient checkpointing trades off computation time for memory savings by recomputing activations
kernel fusion reduces memory bandwidth bottleneck
Can you imagine faster video games without waiting for loading screens?
learning rate warmup does: starts small to avoid early training instability
Why does starting big in learning sometimes lead to chaos?
to standardize: when you need zero mean and unit variance for gradient-based optimization
Why do we need to make data uniform before training a model?
Swipe through 100 ML concepts daily
Open Pocket Polymath