Ever wondered how a computer decides what's more important when sorting through tons of data?
Image: ManfredKloeppel, CC BY 3.0, via Wikimedia Commons
cosine annealing does: lr = lr_min + 0.5(lr_max - lr_min)(1 + cos(πt/T))
Ever wondered how a computer decides what's more important when sorting through tons of data?
Imagine you're trying to quickly find your favorite song in a massive playlist. You want the most popular songs to appear first.
Think of it like tuning a radio dial. You start at a certain frequency (learning rate) and gradually adjust it to find the best signal (optimal learning rate) for your favorite station (model performance).
Example
If you start at a high frequency and keep adjusting until you hit the sweet spot, that's like finding the best learning rate for your model.
Remember this
Cosine annealing smoothly adjusts the learning rate to find the best performance.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Learning to rank
Learning rate cosine annealing formula: learning_rate = learning_rate_initial * 0.5 * (1 + cos(pi * epoch / total_epochs))
Adam has bias correction: divides by (1-β^t) in early steps
Why do we sometimes need to fix mistakes in computer decisions?
Matrix multiplication algorithm
Ever wondered how computers speed up multiplying huge numbers?
temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform
How does adjusting T affect the certainty of choices?
label smoothing does: replaces one-hot [0,0,1,0] with [0.025, 0.025, 0.925, 0.025]
How can a computer learn without being told exactly what to do?
Proximal gradient methods for learning
Why can't we always find the best path in a maze?
Swipe through 100 ML concepts daily
Open Pocket Polymath