most transformer operations are memory-bound, not compute-bound

Why do computers sometimes get tired?

Imagine you're playing a video game on your computer. Suddenly, it starts to lag, and your character moves slower than before.

It's like your computer is running out of energy to think fast. It's not about doing more calculations; it's about remembering things quickly.

Example

When you try to open many apps at once, your computer struggles because it can't remember where everything is.

Remember this

Most computer tasks are memory-bound, meaning they're limited by how fast the computer can remember information, not by how fast it can calculate.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

ReLU and Leaky ReLU

Why do computers sometimes struggle with simple decisions?

Loop nest optimization

Can speeding up your computer make tasks quicker?

Matrix multiplication algorithm

Ever wondered how computers speed up multiplying huge numbers?

Triton differs from CUDA

Why does a super-fast computer sometimes run slower than a regular one?

Flashbulb memory

Flashbulb memories are vivid but not always accurate

fused kernels do

Why wait for a computer to finish one task before starting another?

Swipe through 100 ML concepts daily