LoRA (machine learning)

LoRA uses r << d for efficient adaptation

LoRA leverages low-rank adaptation (r << d) to fine-tune pre-trained models efficiently. This approach reduces the number of parameters that need to be updated, making it computationally less intensive.

Example

In a 768-dimensional weight matrix, LoRA might only adapt a 64-dimensional rank (r << d), significantly reducing the number of parameters compared to full fine-tuning.

Remember this

This efficiency matters because it allows for quicker adaptation and lower resource consumption.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

2024 in hip-hop

LoRA rank r controls model capacity and parameters

Alex Lora Cercos

Alex Lora is a Spanish film director

UMAP is faster than t-SNE

UMAP is faster due to approximate nearest neighbors and cross-entropy optimization

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

LoRA vs full fine-tuning: LoRA trains rank-r adapters (~0.1% params), full FT updates everything

LoRA trains rank-r adapters (~0.1% params), full FT updates everything

the reverse process learns: p_θ(x_{t-1}|x_t)

Ancestral reconstruction extrapolates characteristics back in time

Swipe through 100 ML concepts daily

Open Pocket Polymath