ill-conditioned matrices cause numerical instability: small input changes → large output changes

Ill-conditioned matrices lead to numerical instability

Ill-conditioned matrices are those for which small changes in input can cause large changes in output. This sensitivity can lead to significant errors in numerical computations, making results unreliable.

Example

Consider a matrix A with a condition number close to 1. If we slightly alter an input vector x by adding a small perturbation δx, the resulting output vector Ax can change dramatically, illustrating the instability.

Remember this

Understanding this concept is crucial for developing stable numerical algorithms and ensuring accurate results in scientific computations.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

AdaGrad's learning rate decays to zero

Why does a car's speed drop when it goes uphill?

Eigenvalues and eigenvectors

Eigenvectors are unchanged in direction by a linear transformation

Convolutional neural network

Can a neural network learn too well?

Ordinary least squares

OLS minimizes squared differences

gradient accumulation simulates larger batch sizes without more memory

Can you train a machine like you do with a computer?

to standardize: when you need zero mean and unit variance for gradient-based optimization

Why do we need to make data uniform before training a model?

Swipe through 100 ML concepts daily

Open Pocket Polymath