Gradient points uphill in the direction of steepest increase of f
Gradient
Gradient points uphill in the direction of steepest increase of f
The gradient of a function indicates the direction of steepest ascent, guiding us toward the point of maximum increase.
The gradient vector field transforms like a vector under changes in the coordinate system, maintaining its fundamental properties.
Stationary points, where the gradient is zero, are crucial in optimization as they indicate potential maxima, minima, or saddle points.
Example
For f(x, y) = x^2 + y^2, the gradient ∇f = (2x, 2y) points uphill in the direction of steepest increase.
Remember this
Understanding the gradient direction helps in finding optimal solutions in various applications like machine learning and optimization.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
the momentum term does: v_t = βv_{t-1} + ∇L, accumulates gradient direction
Momentum term accelerates convergence in the gradient direction
saddle points are more common than local minima in high dimensions
Why do some mountains have flat tops instead of peaks or valleys?
Definite matrix
How do we measure the steepness of a hill in all directions?
Matrix calculus
How do we find the steepness of a hill at any point?
AdaGrad's learning rate decays to zero
Why does a car's speed drop when it goes uphill?
non-convex loss landscapes are hard: many local minima and saddle points
Why do some hills have more tricky paths than others?
Swipe through more Machine Learning concepts
Open Pocket Polymath