ReLU and Leaky ReLU

Why do computers sometimes struggle with simple decisions?

Imagine you're trying to decide if you should walk or drive to work. If it's sunny and warm, you might choose to walk. If it's raining or cold, you'd probably drive. But what if you can't decide? That's what happens in computers when they can't make a simple decision.

Computers need rules to help them make decisions. The ReLU function helps by saying, "If the input is positive, keep it; if not, ignore it." This way, computers can make decisions without getting stuck.

Example

If the input is 5 (sunny and warm), ReLU keeps it as 5 (walk). If the input is -3 (raining and cold), ReLU ignores it, and the output is 0 (drive).

Remember this

ReLU helps computers make decisions by ignoring negative inputs, simplifying the choice.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

most transformer operations are memory-bound, not compute-bound

Why do computers sometimes get tired?

Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

Strength reduction

Why do computers sometimes get tired of doing the same math over and over?

History of general-purpose CPUs

Why can't computers talk to each other as fast as we want?

branch prediction does: guesses which way an if-statement will go to keep the pipeline full

How can a computer make decisions faster?

Overlapping subproblems

Ever calculated a huge Fibonacci sequence by hand?

Swipe through 100 ML concepts daily

Open Pocket Polymath