Gradient descent weight update equation: w := w - α * ∇J(w)
Gradient descent
Gradient descent weight update equation: w := w - α * ∇J(w)
Interactive — learn by doing
Gradient descent is an optimization algorithm used to minimize functions by iteratively moving towards the steepest descent direction.
The weight update equation w := w - α * ∇J(w) represents the core of gradient descent, where w is the current weight vector, α (alpha) is the learning rate, and ∇J(w) is the gradient of the cost function J with respect to the weights w.
This equation ensures that the weights are adjusted in the direction that reduces the cost function, leading to convergence towards the minimum value of J.
Example
Suppose we have a cost function J(w) = (w - 3)², the gradient ∇J(w) = 2(w - 3). If our current weight w = 5 and learning rate α = 0.1, the weight update would be w := 5 - 0.1 * 2(5 - 3) = 5 - 0.1 * 4 = 4.6.
Remember this
Understanding the weight update equation is crucial for implementing gradient descent in machine learning models, enabling them to learn and minimize error effectively.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Regression analysis
Linear regression equation: ŷ = β0 + β1X
Stochastic gradient descent
Policy Gradient Theorem Equation
Adam optimizer weight update with m and v terms
Ever wondered how we can predict complex financial markets with high accuracy?
Normalization (machine learning)
L2 normalization equation: x_i' = x_i / ||x||_2
Hessian matrix
The Hessian matrix is denoted by H or ∇²
the momentum term does: v_t = βv_{t-1} + ∇L, accumulates gradient direction
Momentum term accelerates convergence in the gradient direction
Swipe through 100 ML concepts daily
Open Pocket Polymath