Pre-LN transformers are easier to train

Vanishing gradient problem

Residual connections help by allowing gradient flow through the skip connection

Transformer (deep learning)

Transformers use multi-head attention for contextualizing tokens

the over-smoothing problem is in GNNs: deep GNNs make all node features converge

Over-smoothing in GNNs causes node features to converge

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

gradient clipping does: caps gradient norm to prevent exploding gradients

How do deep learning networks avoid getting stuck or going haywire during training?

transformers use LayerNorm not BatchNorm

LayerNorm normalizes across all features, accommodating variable-length sequences unlike BatchNorm, which relies on fixed-size batches