RMSprop uses an exponentially decaying average of squared gradients, unlike AdaGrad's cumulative sum
Image: Brown, J., O.J. Ferrians, Jr., J.A. Heginbottom, and E.S. Melnikov. 1998, revised February 2001. Circum-arctic map of pe, Public domain, via Wikimedia Commons
RMSprop fixes about AdaGrad: uses exponential moving average instead of sum
RMSprop uses an exponentially decaying average of squared gradients, unlike AdaGrad's cumulative sum
Related concepts
AdaGrad does: divides learning rate by sqrt of sum of squared gradients
How do we avoid overshooting in learning?
the β₁ and β₂ hyperparameters control in Adam
β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer
Errors and residuals
Ever wonder how accurate predictions are measured?
Ridge regression uses L2 to shrink coefficients without eliminating them
Why do some roads get smoother and straighter over time?
AdaGrad's learning rate decays to zero
Why does a car's speed drop when it goes uphill?
second-order methods (Newton's) converge faster but are expensive: O(n³) per step
Second-order methods converge faster due to quadratic convergence but are expensive due to O(n³) per iteration
Swipe through more Machine Learning concepts
Open Pocket Polymath