KV-cache stores previously computed outputs to avoid redundant calculations in autoregressive models
Image: BruceBlaus, CC BY 3.0, via Wikimedia Commons
KV-cache reduces redundant computation in autoregressive generation
KV-cache stores previously computed outputs to avoid redundant calculations in autoregressive models
Related concepts
GQA reduces KV-cache memory by the group factor
Ever wondered how websites stay fresh in search results?
Tesla Model Y
Tesla Model Y is the world's best-selling electric vehicle in 2023
the reverse process learns: p_θ(x_{t-1}|x_t)
Ancestral reconstruction extrapolates characteristics back in time
MoE models have more parameters but similar compute cost
MoE models distribute parameters across k experts, reducing active experts' compute cost
CPU cache
L1/L2 cache hierarchy reduces global memory latency
paged attention (vLLM) improves serving throughput
Paged attention (vLLM) improves serving throughput by reducing latency through non-contiguous KV-cache pages, enabling faster data retrieval
Swipe through 100 ML concepts daily
Open Pocket Polymath