Can we train smarter, not harder?
Language model benchmark
Can we train smarter, not harder?
Imagine you're trying to bake a cake with a recipe that calls for precise measurements. You have a kitchen scale that measures in grams, but you only have measuring cups marked in cups. You know the conversion factor, but it's tedious to convert every measurement.
Think of mixed precision training like using measuring cups for some ingredients and grams for others. You don't sacrifice the cake's quality but save time and effort in conversion.
Example
Instead of converting 250 grams of flour to cups, you use cups directly for some ingredients, saving time and reducing the chance of conversion errors.
Remember this
Mixed precision training saves computational resources and time without compromising accuracy.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
mixed precision training does: forward in FP16, accumulate gradients in FP32
Ever wished your phone's camera could take stunning photos even in low light?
to standardize: when you need zero mean and unit variance for gradient-based optimization
Why do we need to make data uniform before training a model?
Fine-tuning (deep learning)
Why can we make a big change with just a tiny tweak?
Neural network (machine learning)
Ever tried adjusting the learning rate like tuning a musical instrument?
Machine learning
Can we teach computers to learn like humans?
Reparameterization trick
Reparameterization trick enables differentiable sampling for VAE training
Swipe through more Machine Learning concepts
Open Pocket Polymath