RMSprop uses an exponentially decaying average of squared gradients, unlike AdaGrad's cumulative sum
Image: Brown, J., O.J. Ferrians, Jr., J.A. Heginbottom, and E.S. Melnikov. 1998, revised February 2001. Circum-arctic map of pe, Public domain, via Wikimedia Commons
RMSprop fixes about AdaGrad: uses exponential moving average instead of sum
RMSprop uses an exponentially decaying average of squared gradients, unlike AdaGrad's cumulative sum
Related concepts
AdaGrad does: divides learning rate by sqrt of sum of squared gradients
How do we avoid overshooting in learning?
the β₁ and β₂ hyperparameters control in Adam
β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer
Ridge regression uses L2 to shrink coefficients without eliminating them
Why do some roads get smoother and straighter over time?
AdaGrad's learning rate decays to zero
Why does a car's speed drop when it goes uphill?
second-order methods (Newton's) converge faster but are expensive: O(n³) per step
Second-order methods converge faster due to quadratic convergence but are expensive due to O(n³) per iteration
to standardize: when you need zero mean and unit variance for gradient-based optimization
Why do we need to make data uniform before training a model?
Swipe through 100 ML concepts daily
Open Pocket Polymath