LoRA vs full fine-tuning: LoRA trains rank-r adapters (~0.1% params), full FT updates everything

Fine-tuning (deep learning)

Why can we make a big change with just a tiny tweak?

2024 in hip-hop

LoRA rank r controls model capacity and parameters

LoRA (machine learning)

LoRA uses r << d for efficient adaptation

Alex Lora Cercos

Alex Lora is a Spanish film director

L1 vs L2 regularization: L1 gives sparsity (feature selection), L2 gives small weights

L1 regularization: L1 = L2 + sparsity; L2 regularization: L2 = L1 + small weights

Pre-LN transformers are easier to train

Pre-LN transformers use residual connections, allowing gradients to flow more smoothly during backpropagation