Why do all friends in a group chat seem to start talking about the same thing after a while?
Image: Mike Tigas from Columbia, MO, United States, CC BY 2.0, via Wikimedia Commons
the over-smoothing problem is in GNNs: deep GNNs make all node features converge
Why do all friends in a group chat seem to start talking about the same thing after a while?
Imagine you're at a party with a group of friends, and everyone starts discussing the same topic after a few minutes. It's like everyone's conversation is getting influenced by the others.
In a group chat, if everyone's talking about the same thing, it's as if their conversations are blending together. This is similar to what's called "over-smoothing" in Graph Neural Networks (GNNs), where node features converge and become indistinguishable.
Example
If you have 5 friends at a party, and after 10 minutes, all of them are talking about the same movie, that's like over-smoothing happening in a GNN.
Remember this
Over-smoothing in GNNs means that after several iterations, all nodes' features become too similar, losing their unique characteristics.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
message passing does in GNNs: each node aggregates features from its neighbors
Can you imagine a social network where your connections influence your opinions?
Convolutional neural network
Can a neural network learn too well?
batch size affects generalization: larger batches find sharper minima
Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates
gradient clipping does: caps gradient norm to prevent exploding gradients
How do deep learning networks avoid getting stuck or going haywire during training?
GCN (Graph Convolutional Network) does: spectral convolution approximated by neighbor averaging
How can we predict a molecule's medical efficacy?
GAT (Graph Attention Network) adds: learned attention weights between neighbors
How do computers learn what to focus on in a complex network?
Swipe through 100 ML concepts daily
Open Pocket Polymath