GPTQ quantization does

AWQ does differently

AWQ selectively retains weights crucial for model performance, unlike traditional quantization

GPTQ vs AWQ: GPTQ uses Hessian-based quantization, AWQ preserves activation-important weights

GPTQ applies Hessian-based quantization, AWQ retains weights crucial for activations

quantization to INT8 doubles throughput

Quantization to INT8 doubles throughput because tensor cores process INT8 2x faster

Vector quantization

How can you store a huge library of books in a tiny closet?

Shannon's source coding theorem: you can't compress below entropy

Can you squeeze endless text into fewer bits without losing anything?

ALiBi allows length extrapolation better than learned position embeddings

ALiBi uses relative positional encoding, avoiding fixed-size embeddings, enabling better handling of variable-length sequences