Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

Imagine you're playing a video game where characters are supposed to be fair, but sometimes they favor one player over another without reason.

If a computer game starts favoring one player, we might need to adjust its rules early on to keep things fair for everyone.

Example

In the first few levels, the game might accidentally give one player twice as many points as others.

Remember this

By dividing by (1-β^t), we correct early mistakes to ensure fairness in the game's outcomes.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

the β₁ and β₂ hyperparameters control in Adam

β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer

ReLU and Leaky ReLU

Why do computers sometimes struggle with simple decisions?

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

Cross-entropy loss penalizes confident wrong predictions more heavily

gradient checkpointing trades: recomputes activations to save memory

Gradient checkpointing trades off computation time for memory savings by recomputing activations

Overlapping subproblems

Ever calculated a huge Fibonacci sequence by hand?

Masking (behavior)

Can you not see what's right in front of you?

Swipe through 100 ML concepts daily

Open Pocket Polymath