AWQ does differently

GPTQ vs AWQ: GPTQ uses Hessian-based quantization, AWQ preserves activation-important weights

GPTQ applies Hessian-based quantization, AWQ retains weights crucial for activations

GPTQ quantization does

Post-training quantization using second-order information for model compression

Federated learning

Ever wonder how to train a smart device without sharing your personal data?

quantization to INT8 doubles throughput

Quantization to INT8 doubles throughput because tensor cores process INT8 2x faster

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

MoE models have more parameters but similar compute cost

MoE models distribute parameters across k experts, reducing active experts' compute cost