multi-query attention (MQA) is

grouped query attention (GQA) does

GQA shares KV heads across multiple Q heads for efficient parameter usage

Write the multi-head attention formula: MultiHead(Q,K,V) = Concat(head_1,...,head_h)W^O

Ever wondered how machines understand the importance of words in a sentence?

self-attention: Attention(Q,K,V) = softmax(QK^T/√d_k)V

Ever wondered how computers understand what's important in a sentence?

paged attention (vLLM) improves serving throughput

Paged attention (vLLM) improves serving throughput by reducing latency through non-contiguous KV-cache pages, enabling faster data retrieval

Pre-LN

Why is normalizing data like tuning instruments before a concert?

Mixture of experts

Mixture of experts (MoE) divides problem space into homogeneous regions