Gradient points uphill in the direction of steepest increase of f

Gradient

Gradient points uphill in the direction of steepest increase of f

The gradient of a function indicates the direction of steepest ascent, guiding us toward the point of maximum increase.

The gradient vector field transforms like a vector under changes in the coordinate system, maintaining its fundamental properties.

Stationary points, where the gradient is zero, are crucial in optimization as they indicate potential maxima, minima, or saddle points.

Example

For f(x, y) = x^2 + y^2, the gradient ∇f = (2x, 2y) points uphill in the direction of steepest increase.

Remember this

Understanding the gradient direction helps in finding optimal solutions in various applications like machine learning and optimization.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

the momentum term does: v_t = βv_{t-1} + ∇L, accumulates gradient direction

Momentum term accelerates convergence in the gradient direction

saddle points are more common than local minima in high dimensions

Why do some mountains have flat tops instead of peaks or valleys?

AdaGrad's learning rate decays to zero

Why does a car's speed drop when it goes uphill?

non-convex loss landscapes are hard: many local minima and saddle points

Why do some hills have more tricky paths than others?

Tangent space

Tangent space at a point represents all possible velocity vectors

Geodesics on an ellipsoid

Geodesics are the shortest paths on a curved surface

Swipe through 100 ML concepts daily