Paged attention (vLLM) improves serving throughput by reducing latency through non-contiguous KV-cache pages, enabling faster data retrieval
Image: Daniel Voigt Godoy, CC BY 4.0, via Wikimedia Commons
paged attention (vLLM) improves serving throughput
Paged attention (vLLM) improves serving throughput by reducing latency through non-contiguous KV-cache pages, enabling faster data retrieval
Related concepts
GQA reduces KV-cache memory by the group factor
Ever wondered how websites stay fresh in search results?
grouped query attention (GQA) does
GQA shares KV heads across multiple Q heads for efficient parameter usage
Flashbulb memory
Flashbulb memories are vivid but not always accurate
CPU cache
L1/L2 cache hierarchy reduces global memory latency
Attention (machine learning)
Why do we sometimes zoom in faster on a scene?
Loop nest optimization
Can speeding up your computer make tasks quicker?
Swipe through 100 ML concepts daily
Open Pocket Polymath