Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

Imagine you're playing a video game where characters are supposed to be fair, but sometimes they favor one player over another without reason.

If a computer game starts favoring one player, we might need to adjust its rules early on to keep things fair for everyone.

Example

In the first few levels, the game might accidentally give one player twice as many points as others.

Remember this

By dividing by (1-β^t), we correct early mistakes to ensure fairness in the game's outcomes.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

the β₁ and β₂ hyperparameters control in Adam

β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer

ReLU and Leaky ReLU

Why do computers sometimes struggle with simple decisions?

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

How wrong predictions hurt more than they're supposed to

gradient checkpointing trades: recomputes activations to save memory

Gradient checkpointing trades off computation time for memory savings by recomputing activations

Strength reduction

Why do computers sometimes get tired of doing the same math over and over?

ill-conditioned matrices cause numerical instability: small input changes → large output changes

Small changes can lead to huge errors in math

Swipe through 100 ML concepts daily

Open Pocket Polymath