Second-order methods converge faster due to quadratic convergence but are expensive due to O(n³) per iteration
Image: Hans Hillewaert, CC BY-SA 4.0, via Wikimedia Commons
second-order methods (Newton's) converge faster but are expensive: O(n³) per step
Second-order methods converge faster due to quadratic convergence but are expensive due to O(n³) per iteration
Related concepts
Linear multistep method
Why does skipping steps save time but lose accuracy?
approximation algorithms guarantee: solution within factor α of optimal
Can we always find the best solution quickly?
Overlapping subproblems
Ever calculated a huge Fibonacci sequence by hand?
the momentum term does: v_t = βv_{t-1} + ∇L, accumulates gradient direction
Momentum term accelerates convergence in the gradient direction
ill-conditioned matrices cause numerical instability: small input changes → large output changes
Small changes can lead to huge errors in math
Strength reduction
Why do computers sometimes get tired of doing the same math over and over?
Swipe through 100 ML concepts daily
Open Pocket Polymath