ReLU and Leaky ReLU

Why do computers sometimes struggle with simple decisions?

Imagine you're trying to decide if you should walk or drive to work. If it's sunny and warm, you might choose to walk. If it's raining or cold, you'd probably drive. But what if you can't decide? That's what happens in computers when they can't make a simple decision.

Computers need rules to help them make decisions. The ReLU function helps by saying, "If the input is positive, keep it; if not, ignore it." This way, computers can make decisions without getting stuck.

Example

If the input is 5 (sunny and warm), ReLU keeps it as 5 (walk). If the input is -3 (raining and cold), ReLU ignores it, and the output is 0 (drive).

Remember this

ReLU helps computers make decisions by ignoring negative inputs, simplifying the choice.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

most transformer operations are memory-bound, not compute-bound

Why do computers sometimes get tired?

Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

Strength reduction

Why do computers sometimes get tired of doing the same math over and over?

Attempts to overturn the 2020 United States presidential election

Why can computers do many things at once?

History of general-purpose CPUs

Why can't computers talk to each other as fast as we want?

Likelihood function

Why can't we just guess probabilities correctly all the time?

Swipe through more Machine Learning concepts

Open Pocket Polymath