most transformer operations are memory-bound, not compute-bound

Why do computers sometimes get tired?

Imagine you're playing a video game on your computer. Suddenly, it starts to lag, and your character moves slower than before.

It's like your computer is running out of energy to think fast. It's not about doing more calculations; it's about remembering things quickly.

Example

When you try to open many apps at once, your computer struggles because it can't remember where everything is.

Remember this

Most computer tasks are memory-bound, meaning they're limited by how fast the computer can remember information, not by how fast it can calculate.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Strength reduction

Why do computers sometimes get tired of doing the same math over and over?

ReLU and Leaky ReLU

Why do computers sometimes struggle with simple decisions?

Loop nest optimization

Can speeding up your computer make tasks quicker?

2022 in science

Why do Transformers sometimes seem to 'ignore' irrelevant parts of the input?

Matrix multiplication algorithm

Ever wondered how computers speed up multiplying huge numbers?

Triton differs from CUDA

Why does a super-fast computer sometimes run slower than a regular one?

Swipe through 100 ML concepts daily