kernel fusion reduces memory bandwidth bottleneck

Can you imagine faster video games without waiting for loading screens?

Imagine you're playing an online game on your computer. Every time you switch levels, the game pauses and waits for the new level to load, making you wait longer and less fun.

Think of your computer as a busy highway. Right now, it's like cars (data) taking separate roads (memory) to get to the destination (your GPU). If all cars took the same road, traffic jams (slowdowns) would be less frequent.

Example

Currently, each level switch sends data down different memory paths. If all data went through one path, loading times would drop.

Remember this

Kernel fusion combines separate tasks into one, reducing the number of memory paths used and speeding up loading times.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

fused kernels do

Why wait for a computer to finish one task before starting another?

Loop nest optimization

Can speeding up your computer make tasks quicker?

Triton auto-tunes BLOCK_SIZE: different sizes optimize for different hardware

Rate-distortion optimization balances video quality and file size

CPU cache

L1/L2 cache hierarchy reduces global memory latency

operator fusion does at the compiler level: merges adjacent ops to reduce memory traffic

Compiler optimizations merge adjacent operations to reduce memory traffic

Von Neumann architecture

CPU must fetch both data and instructions from memory

Swipe through 100 ML concepts daily

Open Pocket Polymath