Chinchilla scaling law: optimal model size scales linearly with compute budget
Neural scaling law
Chinchilla scaling law: optimal model size scales linearly with compute budget
The Chinchilla scaling law demonstrates a direct relationship between the size of a neural network model and the available compute budget. This scaling law is particularly relevant for optimizing resource allocation in machine learning tasks.
Example
In a practical scenario, if a machine learning team has a fixed compute budget, they can increase the model size proportionally to maximize performance, as suggested by the Chinchilla scaling law.
Remember this
Understanding this scaling law helps in efficiently utilizing compute resources to achieve optimal model performance.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
MoE models have more parameters but similar compute cost
MoE models distribute parameters across k experts, reducing active experts' compute cost
2024 in hip-hop
LoRA rank r controls model capacity and parameters
to standardize: when you need zero mean and unit variance for gradient-based optimization
Why do we need to make data uniform before training a model?
the determinant tells you about volume scaling under a linear transformation
The determinant of a matrix representing a linear transformation indicates the factor by which volumes are scaled
the compute-optimal training ratio is: roughly 20 tokens per parameter
Compute-optimal training ratio: roughly 20 tokens per parameter
Manifold hypothesis
High-dimensional data lies on lower-dimensional manifolds
Swipe through 100 ML concepts daily
Open Pocket Polymath