label smoothing does: replaces one-hot [0,0,1,0] with [0.025, 0.025, 0.925, 0.025]

How can a computer learn without being told exactly what to do?

Imagine you're trying to teach a friend to recognize different fruits by sight alone. You show them pictures of apples, oranges, and bananas, but instead of telling them which is which, you give them a hint that one fruit is mostly red, another is mostly orange, and the last one is mostly yellow.

So, you don't just show them the fruits; you also give them a little bit of information about each fruit. This helps them guess the fruit's identity more accurately. The technique we're talking about is called label smoothing.

Example

If you usually tell your friend that an apple is 100% red (1,0,0,0), you might instead say it's 75% red (0.75,0,0.25,0) to help them learn better.

Remember this

Label smoothing helps computers learn by giving them a bit of uncertainty, making them better at guessing the right answers.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Effect size

Cohen's D benchmarks: 0.2 = small, 0.5 = medium, 0.8 = large effect

Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

cosine annealing does: lr = lr_min + 0.5(lr_max - lr_min)(1 + cos(πt/T))

Ever wondered how a computer decides what's more important when sorting through tons of data?

300-dim word2vec encodes: trained on word co-occurrence with skip-gram window

Ever wondered how computers understand words?

a parity check bit does: detects single-bit errors by making total 1s even/odd

Parity bit makes total 1s even/odd

TF-IDF scoring

TF-IDF = (Term Frequency) * (Inverse Document Frequency)

Swipe through 100 ML concepts daily

Open Pocket Polymath