What is GPTQ?

Definition

GPTQ (Generalized Post-Training Quantization) is a post-training quantization method that compresses model weights to 4-bit precision while preserving substantially more quality than naive uniform quantization. The technique works layer-by-layer: for each weight matrix, GPTQ uses information from a small calibration dataset to compute second-order statistics about how quantization error in different weights propagates through the layer, then chooses quantization values that minimize total layer error rather than per-weight error.

The practical result is that a GPTQ-quantized 4-bit model typically retains 95-99% of the original FP16 model's accuracy on standard benchmarks while using approximately 4x less memory. GPTQ is widely supported across inference frameworks — vLLM, TensorRT-LLM, ExLlamaV2, and others all consume GPTQ-quantized models directly. It is a common alternative to AWQ, with the relative performance varying by model family.

Why It Matters

Quantization is the difference between a model that fits on your hardware and one that doesn't. GPTQ produces high-quality 4-bit quantized models that are widely deployable — the format is well-supported across inference frameworks, and many community-quantized GPTQ versions of popular open-weight models are available on Hugging Face. For teams running inference on consumer GPUs or trying to fit more concurrent requests on server hardware, GPTQ is a standard tool.

Key Takeaways

• GPTQ is a post-training 4-bit weight quantization method — no fine-tuning required
• Uses calibration-data second-order statistics to minimize layer-level quantization error
• Typically retains 95-99% of FP16 accuracy at ~4x memory reduction
• Widely supported across vLLM, TensorRT-LLM, ExLlamaV2, and other inference frameworks
• Common alternative to AWQ — relative quality varies by model family