How do deep learning networks avoid getting stuck or going haywire during training?
Image: Ulli Purwin, CC BY 3.0, via Wikimedia Commons
gradient clipping does: caps gradient norm to prevent exploding gradients
How do deep learning networks avoid getting stuck or going haywire during training?
Imagine you're learning to play a complex musical instrument. As you practice, some notes become harder to hit because your fingers get tired, while others are easier. If you don't stop practicing, your fingers might cramp up, making it impossible to play any notes at all.
The idea is like setting a limit on how tired your fingers can get. In deep learning, we set a limit on how big the changes to the network's parameters can be during training. This limit prevents the learning process from getting overwhelmed and failing.
Example
Think of your fingers as weights in a neural network. If you practice too hard without breaks, they might cramp up (explode). By taking short breaks (clipping), you keep them from getting too tired (exploding).
Remember this
Gradient clipping is like taking breaks to prevent your fingers from cramping up while learning to play an instrument.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Vanishing gradient problem
Residual connections help by allowing gradient flow through the skip connection
gradient accumulation simulates larger batch sizes without more memory
Can you train a machine like you do with a computer?
the over-smoothing problem is in GNNs: deep GNNs make all node features converge
Over-smoothing in GNNs causes node features to converge
gradient checkpointing trades: recomputes activations to save memory
Gradient checkpointing trades off computation time for memory savings by recomputing activations
Pre-LN transformers are easier to train
Pre-LN transformers use residual connections, allowing gradients to flow more smoothly during backpropagation
Convolutional neural network
Can a neural network learn too well?
Swipe through 100 ML concepts daily
Open Pocket Polymath