β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer
Image: Newspaper Enterprise Association, Public domain, via Wikimedia Commons
the β₁ and β₂ hyperparameters control in Adam
β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer
Related concepts
Adam has bias correction: divides by (1-β^t) in early steps
Why do we sometimes need to fix mistakes in computer decisions?
Adam vs SGD: Adam adapts per-parameter rates, SGD often generalizes better with tuning
Adam adjusts learning rates per-parameter, SGD generalizes better with tuning
Adam combines momentum and RMSprop: adapts per-parameter learning rates
Ever wondered how a car adjusts its speed for smooth turns?
gradient checkpointing trades: recomputes activations to save memory
Gradient checkpointing trades off computation time for memory savings by recomputing activations
quantization to INT8 doubles throughput
Quantization to INT8 doubles throughput because tensor cores process INT8 2x faster
DDIM does: deterministic sampling for faster generation with fewer steps
DDIM accelerates image generation by deterministically sampling intermediate steps
Swipe through 100 ML concepts daily
Open Pocket Polymath