2024 in hip-hop

LoRA rank r controls model capacity and parameters

In LoRA, the rank r determines the model's capacity and the number of parameters. A higher rank results in a model with greater capacity and more parameters.

Example

A LoRA model with rank r=32 will have more capacity and parameters compared to a model with rank r=16.

Remember this

Understanding the relationship between rank r and model capacity is crucial for optimizing performance in LoRA models.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

LoRA (machine learning)

LoRA uses r << d for efficient adaptation

Alex Lora Cercos

Alex Lora is a Spanish film director

LoRA vs full fine-tuning: LoRA trains rank-r adapters (~0.1% params), full FT updates everything

LoRA trains rank-r adapters (~0.1% params), full FT updates everything

MoE models have more parameters but similar compute cost

MoE models distribute parameters across k experts, reducing active experts' compute cost

Neural scaling law

Chinchilla scaling law: optimal model size scales linearly with compute budget

PCA vs t-SNE: PCA preserves global variance linearly, t-SNE preserves local structure nonlinearly

How can we teach computers to understand what we like?

Swipe through 100 ML concepts daily

Open Pocket Polymath