second-order methods (Newton's) converge faster but are expensive: O(n³) per step

Finite element method

Runge-Kutta method improves Euler by providing higher-order accuracy with k₁,k₂,k₃,k₄

approximation algorithms guarantee: solution within factor α of optimal

Can we always find the best solution quickly?

Overlapping subproblems

Ever calculated a huge Fibonacci sequence by hand?

iterative methods (CG, GMRES) do: solve Ax=b without explicitly inverting A

Iterative methods solve Ax=b without explicitly inverting A

the momentum term does: v_t = βv_{t-1} + ∇L, accumulates gradient direction

Momentum term accelerates convergence in the gradient direction

Greedy vs dynamic programming: greedy makes locally optimal choices, DP considers all subproblems

Greedy: locally optimal choices; DP: considers all subproblems