Gradient descent weight update equation: w := w - α * ∇J(w)
Gradient descent
Gradient descent weight update equation: w := w - α * ∇J(w)
Interactive — learn by doing
Gradient descent is an optimization algorithm used to minimize functions by iteratively moving towards the steepest descent direction.
The weight update equation w := w - α * ∇J(w) represents the core of gradient descent, where w is the current weight vector, α (alpha) is the learning rate, and ∇J(w) is the gradient of the cost function J with respect to the weights w.
This equation ensures that the weights are adjusted in the direction that reduces the cost function, leading to convergence towards the minimum value of J.
Example
Suppose we have a cost function J(w) = (w - 3)², the gradient ∇J(w) = 2(w - 3). If our current weight w = 5 and learning rate α = 0.1, the weight update would be w := 5 - 0.1 * 2(5 - 3) = 5 - 0.1 * 4 = 4.6.
Remember this
Understanding the weight update equation is crucial for implementing gradient descent in machine learning models, enabling them to learn and minimize error effectively.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Backpropagation
How can we improve a recipe step-by-step?
Regression analysis
Linear regression equation: ŷ = β0 + β1X
Stochastic gradient descent
Policy Gradient Theorem Equation
Adam optimizer weight update with m and v terms
Ever wondered how we can predict complex financial markets with high accuracy?
Normalization (machine learning)
L2 normalization equation: x_i' = x_i / ||x||_2
Matrix calculus
How do we find the steepness of a hill at any point?
Swipe through more Machine Learning concepts
Open Pocket Polymath