Pre-LN transformers are easier to train

Vanishing gradient problem

Residual connections help by allowing gradient flow through the skip connection

Transformer (deep learning)

Transformers use multi-head attention for contextualizing tokens

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

2022 in science

Why do Transformers sometimes seem to 'ignore' irrelevant parts of the input?

gradient clipping does: caps gradient norm to prevent exploding gradients

How do deep learning networks avoid getting stuck or going haywire during training?

transformers use LayerNorm not BatchNorm

LayerNorm normalizes across all features, accommodating variable-length sequences unlike BatchNorm, which relies on fixed-size batches