iterative methods (CG, GMRES) do: solve Ax=b without explicitly inverting A

How can we find answers without solving problems head-on?

Imagine you're trying to figure out the best route for a road trip without actually driving the entire distance. You need to find the best way to get from point A to point B efficiently.

Instead of driving every possible route, you can start with a rough guess and then make small adjustments based on feedback, like traffic updates, to get closer to the best route.

Example

You guess the first part of your route is 10 miles north, then after checking traffic, you adjust to 9.5 miles north.

Remember this

This is the idea behind iterative methods like CG and GMRES, which refine solutions step by step without needing to solve the problem directly.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Linear multistep method

Why does skipping steps save time but lose accuracy?

natural gradient descent does: preconditions with inverse Fisher matrix

Ever wondered how to climb a steep mountain without measuring every step?

approximation algorithms guarantee: solution within factor α of optimal

Can we always find the best solution quickly?

Ordinary least squares

OLS minimizes squared differences

second-order methods (Newton's) converge faster but are expensive: O(n³) per step

Second-order methods converge faster due to quadratic convergence but are expensive due to O(n³) per iteration

Master theorem (analysis of algorithms)

Master theorem solves T(n) = aT(n/b) + f(n) recurrences

Swipe through 100 ML concepts daily

Open Pocket Polymath