Pre-LN

Why is normalizing data like tuning instruments before a concert?

Imagine you're cooking with ingredients measured in different units, like cups and teaspoons. It would be chaotic trying to mix them without a common measuring system.

Normalization in machine learning is like converting cups to teaspoons so all ingredients mix smoothly. It's about making sure data from different sources plays nicely together in a model.

Example

A recipe calls for 2 cups of sugar and 1 teaspoon of salt. Converting cups to teaspoons (1 cup = 48 teaspoons), you get 96 teaspoons of sugar and 1 teaspoon of salt, making it easier to combine them.

Remember this

Normalization ensures different data features work together seamlessly in machine learning models.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

to use an RNN/LSTM: for sequential data where order matters (mostly replaced by transformers)

Why do we remember stories better when they have a clear beginning, middle, and end?

Batch norm vs layer norm: BN across batch, LN across features

Batch norm (BN) normalizes across batch, layer norm (LN) normalizes across features; LN handles variable-length sequences

structured pruning removes: entire filters or attention heads, not individual weights

Why do we sometimes remove parts of a neural network to improve its performance?

paged attention (vLLM) improves serving throughput

Paged attention (vLLM) improves serving throughput by reducing latency through non-contiguous KV-cache pages, enabling faster data retrieval

Pre-LN transformers are easier to train

Pre-LN transformers use residual connections, allowing gradients to flow more smoothly during backpropagation

to normalize features: when features have different scales and you use distance-based methods

Why do some things need to be adjusted to compare fairly?

Swipe through 100 ML concepts daily

Open Pocket Polymath