L1 vs L2 regularization: L1 gives sparsity (feature selection), L2 gives small weights

Regularization (mathematics)

L1 regularization results in sparse solutions

LASSO uses L1 to do feature selection by driving coefficients to exactly zero

Why do some numbers disappear when solving complex problems?

Batch norm vs layer norm: BN across batch, LN across features

Batch norm (BN) normalizes across batch, layer norm (LN) normalizes across features; LN handles variable-length sequences

to normalize features: when features have different scales and you use distance-based methods

Why do some things need to be adjusted to compare fairly?

Boosting (machine learning)

Boosting reduces bias in ML models

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates