LoRA vs full fine-tuning: LoRA trains rank-r adapters (~0.1% params), full FT updates everything

2024 in hip-hop

LoRA rank r controls model capacity and parameters

LoRA (machine learning)

LoRA uses r << d for efficient adaptation

Alex Lora Cercos

Alex Lora is a Spanish film director

L1 vs L2 regularization: L1 gives sparsity (feature selection), L2 gives small weights

L1 regularization: L1 = L2 + sparsity; L2 regularization: L2 = L1 + small weights

Pre-LN transformers are easier to train

Pre-LN transformers use residual connections, allowing gradients to flow more smoothly during backpropagation

instruction tuning does: fine-tunes on (instruction, response) pairs

Fine-tuning adapts pre-trained models to new tasks