Matrix calculus

How do we find the steepness of a hill at any point?

Imagine you're hiking and want to know how steep the hill is at your current location to decide if you should climb or walk around.

Think of the hill as a function that tells you the height at any point. To find the steepness at your location, we look at how the height changes as you move a tiny bit in different directions. This is like looking at the slope of the hill in every direction around you.

Example

If you're standing at a point on the hill, and moving 1 meter north changes your height by 0.5 meters, while moving 1 meter east changes your height by 0.2 meters, you're looking at different slopes in different directions.

Remember this

The Hessian matrix captures all these slopes at once, showing how the hill's steepness changes in every direction around your current location.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Definite matrix

How do we measure the steepness of a hill in all directions?

Hessian matrix

The Hessian matrix is denoted by H or ∇²

Backpropagation

How can we improve a recipe step-by-step?

Lagrangian L(x,λ) = f(x) - λg(x)

How do we find the path an object naturally takes?

natural gradient descent does: preconditions with inverse Fisher matrix

Ever wondered how to climb a steep mountain without measuring every step?

Gradient descent

Gradient descent weight update equation: w := w - α * ∇J(w)

Swipe through more Machine Learning concepts

Open Pocket Polymath