second-order methods (Newton's) converge faster but are expensive: O(n³) per step

Linear multistep method

Why does skipping steps save time but lose accuracy?

approximation algorithms guarantee: solution within factor α of optimal

Can we always find the best solution quickly?

Overlapping subproblems

Ever calculated a huge Fibonacci sequence by hand?

Markov chain

Why does flipping a coin repeatedly get predictable?

the momentum term does: v_t = βv_{t-1} + ∇L, accumulates gradient direction

Momentum term accelerates convergence in the gradient direction

ill-conditioned matrices cause numerical instability: small input changes → large output changes

Small changes can lead to huge errors in math