MoE models have more parameters but similar compute cost

Mixture of experts

Mixture of experts (MoE) divides problem space into homogeneous regions

load balancing loss is needed in MoE

Can one expert handle all tasks perfectly?

AWQ does differently

AWQ selectively retains weights crucial for model performance, unlike traditional quantization

KV-cache reduces redundant computation in autoregressive generation

KV-cache stores previously computed outputs to avoid redundant calculations in autoregressive models

Tesla Model Y

Tesla Model Y is the world's best-selling electric vehicle in 2023

Graduate Aptitude Test in Engineering

GATE exam assesses engineering and science undergraduate subjects for postgraduate admissions in India