ReLU and Leaky ReLU

Why do computers sometimes struggle with simple decisions?

Imagine you're trying to decide if you should walk or drive to work. If it's sunny and warm, you might choose to walk. If it's raining or cold, you'd probably drive. But what if you can't decide? That's what happens in computers when they can't make a simple decision.

Computers need rules to help them make decisions. The ReLU function helps by saying, "If the input is positive, keep it; if not, ignore it." This way, computers can make decisions without getting stuck.

Example

If the input is 5 (sunny and warm), ReLU keeps it as 5 (walk). If the input is -3 (raining and cold), ReLU ignores it, and the output is 0 (drive).

Remember this

ReLU helps computers make decisions by ignoring negative inputs, simplifying the choice.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

most transformer operations are memory-bound, not compute-bound

Why do computers sometimes get tired?

Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

Overlapping subproblems

Ever calculated a huge Fibonacci sequence by hand?

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

Halting problem

Alan Turing proved the halting problem is undecidable

Triton differs from CUDA

Why does a super-fast computer sometimes run slower than a regular one?

Swipe through 100 ML concepts daily

Open Pocket Polymath