Adam optimizer weight update with m and v terms

Ever wondered how we can predict complex financial markets with high accuracy?

Imagine trying to forecast the price of a stock that fluctuates wildly due to countless unpredictable factors.

Picture a stock price as a ball rolling down a hill. The Adam optimizer helps us predict where the ball will land next by adjusting its path based on past movements (m and v terms).

Example

If the ball's current position is 10 units down the hill and it has moved 2 units to the left (m term) and 1 unit to the right (v term) in the past, the optimizer adjusts the ball's path accordingly.

Remember this

The Adam optimizer's m and v terms help us fine-tune predictions by learning from previous adjustments.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Gradient descent

Gradient descent weight update equation: w := w - α * ∇J(w)

Write the Bellman equation for reinforcement learning

Predicting tomorrow's weather with today's clues

to standardize: when you need zero mean and unit variance for gradient-based optimization

Why do we need to make data uniform before training a model?

Adam vs SGD: Adam adapts per-parameter rates, SGD often generalizes better with tuning

Adam adjusts learning rates per-parameter, SGD generalizes better with tuning

Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

Swipe through 100 ML concepts daily

Open Pocket Polymath