Warp divergence causes threads to execute non-uniformly, leading to idle cycles and reduced throughput
Image: Robert M. Lavinsky, CC BY-SA 3.0, via Wikimedia Commons
warp divergence kills performance
Warp divergence causes threads to execute non-uniformly, leading to idle cycles and reduced throughput
Related concepts
Loop nest optimization
Can speeding up your computer make tasks quicker?
occupancy means in GPU programming
Occupancy = Active Warps / Max Warps
Triton differs from CUDA
Why does a super-fast computer sometimes run slower than a regular one?
Flashbulb memory
Flashbulb memories are vivid but not always accurate
loop unrolling does: trades code size for reduced loop overhead
Loop unrolling optimizes execution speed by reducing loop control instructions
gradient checkpointing trades: recomputes activations to save memory
Gradient checkpointing trades off computation time for memory savings by recomputing activations
Swipe through 100 ML concepts daily
Open Pocket Polymath