β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer
Image: Newspaper Enterprise Association, Public domain, via Wikimedia Commons
the β₁ and β₂ hyperparameters control in Adam
β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer
Related concepts
Adam has bias correction: divides by (1-β^t) in early steps
Why do we sometimes need to fix mistakes in computer decisions?
Adam vs SGD: Adam adapts per-parameter rates, SGD often generalizes better with tuning
Adam adjusts learning rates per-parameter, SGD generalizes better with tuning
Adam combines momentum and RMSprop: adapts per-parameter learning rates
Ever wondered how a car adjusts its speed for smooth turns?
gradient checkpointing trades: recomputes activations to save memory
Gradient checkpointing trades off computation time for memory savings by recomputing activations
2022 in science
Why do Transformers sometimes seem to 'ignore' irrelevant parts of the input?
quantization to INT8 doubles throughput
Quantization to INT8 doubles throughput because tensor cores process INT8 2x faster
Swipe through more Machine Learning concepts
Open Pocket Polymath