the lottery ticket hypothesis says: sparse subnetworks can match full network performance

Can a small part of a puzzle fit perfectly into its place by chance?

Imagine you're trying to find a perfect fit for a missing piece in a complex jigsaw puzzle. You have countless pieces to try, but you're looking for the one that fits perfectly without needing to rearrange the rest.

Picture a dense neural network as a massive jigsaw puzzle. The lottery ticket hypothesis suggests that within this vast puzzle, there's a small, randomly chosen piece that can fit perfectly into its place, matching the puzzle's overall picture without needing to rearrange the entire puzzle.

Example

You have a 1000-piece puzzle. Instead of trying every piece to find the perfect fit, you randomly select a small group of pieces (like a subnetwork) and test them individually. Surprisingly, one of these small groups contains the perfect piece that fits perfectly.

Remember this

The core insight is that within a large, complex system (like a neural network), there's a chance that a small, randomly chosen subset (a sparse subnetwork) can perform as well as the whole system.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

dropout works as regularization: it approximates an ensemble of subnetworks

Why does turning off neurons randomly help a brainy computer learn better?

the curse of dimensionality makes nearest neighbor search unreliable

Why can't we find our friends easily as we move to a city with more and more neighborhoods?

message passing does in GNNs: each node aggregates features from its neighbors

Nodes in GNNs aggregate features from neighbors

GraphSAGE does: samples and aggregates a fixed-size neighborhood

GraphSAGE samples and aggregates a fixed-size neighborhood

Chebyshev's inequality

Chebyshev's inequality limits the probability of deviation from the mean

Mixture of experts

Mixture of experts (MoE) divides problem space into homogeneous regions

Swipe through 100 ML concepts daily

Open Pocket Polymath