AWQ does differently

GPTQ vs AWQ: GPTQ uses Hessian-based quantization, AWQ preserves activation-important weights

GPTQ applies Hessian-based quantization, AWQ retains weights crucial for activations

GPTQ quantization does

Post-training quantization using second-order information for model compression

quantization to INT8 doubles throughput

Quantization to INT8 doubles throughput because tensor cores process INT8 2x faster

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

MoE models have more parameters but similar compute cost

MoE models distribute parameters across k experts, reducing active experts' compute cost

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates