the momentum term does: v_t = βv_{t-1} + ∇L, accumulates gradient direction

Gradient

Gradient points uphill in the direction of steepest increase of f

Divergence

How does air flow change volume?

AdaGrad's learning rate decays to zero

Why does a car's speed drop when it goes uphill?

gradient clipping does: caps gradient norm to prevent exploding gradients

How do deep learning networks avoid getting stuck or going haywire during training?

second-order methods (Newton's) converge faster but are expensive: O(n³) per step

Second-order methods converge faster due to quadratic convergence but are expensive due to O(n³) per iteration

Diffusion model

q(x_t|x_{t-1}) adds Gaussian noise at each step