structured pruning removes: entire filters or attention heads, not individual weights

Why do we sometimes remove parts of a neural network to improve its performance?

Imagine you're organizing a cluttered room. You can't throw away every single item; instead, you decide to remove some boxes or shelves to make the space more efficient.

In a neural network, instead of deleting individual weights, we remove entire filters or attention heads that aren't contributing much to the learning process, making the model more efficient.

Example

If you have 10 boxes (filters) and you find that 3 of them are empty, you might decide to remove those 3 boxes entirely.

Remember this

Pruning entire filters or attention heads can lead to a more efficient and faster neural network without significantly impacting its performance.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Convolutional neural network

Can a neural network learn too well?

dropout works as regularization: it approximates an ensemble of subnetworks

Why does turning off neurons randomly help a brainy computer learn better?

Pre-LN

Why is normalizing data like tuning instruments before a concert?

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

soft targets carry more information than hard labels: they encode class similarities

Why do some learning methods need to explore more than others?

Vanishing gradient problem

Residual connections help by allowing gradient flow through the skip connection

Swipe through 100 ML concepts daily

Open Pocket Polymath