Vanishing gradient problem

Residual connections help by allowing gradient flow through the skip connection

Residual connections, also known as skip connections, enable gradients to bypass certain layers in a neural network. This helps mitigate the vanishing gradient problem by providing an alternative path for the gradient flow, ensuring that earlier layers receive sufficient gradient updates.

Example

In a deep neural network, a residual block might consist of a series of convolutional layers followed by a skip connection that adds the input of the block to its output. This allows the gradients to directly flow through the skip connection, bypassing the intermediate layers.

Remember this

Residual connections are crucial for training deep neural networks effectively, as they help maintain stable gradient magnitudes throughout the network.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Pre-LN transformers are easier to train

Pre-LN transformers use residual connections, allowing gradients to flow more smoothly during backpropagation

gradient checkpointing trades: recomputes activations to save memory

Gradient checkpointing trades off computation time for memory savings by recomputing activations

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

gradient clipping does: caps gradient norm to prevent exploding gradients

How do deep learning networks avoid getting stuck or going haywire during training?

dropout works as regularization: it approximates an ensemble of subnetworks

Why does turning off neurons randomly help a brainy computer learn better?

gradient accumulation simulates larger batch sizes without more memory

Can you train a machine like you do with a computer?

Swipe through 100 ML concepts daily

Open Pocket Polymath