Backpropagation

How can we improve a recipe step-by-step?

Imagine you're adjusting a recipe based on taste feedback. Each ingredient's amount affects the final taste, and you want to know which ingredient to tweak for the best result.

Think of adjusting the recipe like finding the steepest slope on a hill. You're trying to get the highest taste (the peak) by changing ingredients (the slope). The technical term for this is calculating the gradient of the loss function with respect to the weights.

Example

If increasing the amount of salt by 1 gram improves taste by 2 units, the gradient tells us how much better the taste will get for each gram of salt added.

Remember this

The gradient shows you the direction and steepness of the best way to improve your recipe.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Gradient descent

Gradient descent weight update equation: w := w - α * ∇J(w)

Matrix calculus

How do we find the steepness of a hill at any point?

Stochastic gradient descent

Policy Gradient Theorem Equation

Pre-LN transformers are easier to train

Pre-LN transformers use residual connections, allowing gradients to flow more smoothly during backpropagation

Write the contrastive loss function for SimCLR

Contrastive loss function: L = (1/2N) Σ [max(0, margin - y_i * (z_i - z_j))^2 + max(0, y_i * (z_i - z_j) - margin)^2]

natural gradient descent does: preconditions with inverse Fisher matrix

Ever wondered how to climb a steep mountain without measuring every step?

Swipe through more Machine Learning concepts

Open Pocket Polymath