What is Quantization?

Definition

Quantization is a model compression technique that converts the high-precision floating-point numbers used during training into lower-precision representations for inference. During training, neural networks typically use 16-bit (FP16) or 32-bit (FP32) floating-point weights to maintain the gradient resolution needed for learning. However, once a model is trained, much of that precision is redundant for generating predictions. Quantization exploits this redundancy by mapping weights — and sometimes activations — to smaller data types such as 8-bit integers (INT8) or even 4-bit integers (INT4).

There are two broad families of quantization. Post-training quantization (PTQ) takes a fully trained model and converts its weights after the fact, sometimes using a small calibration dataset to minimize accuracy loss. Quantization-aware training (QAT), by contrast, simulates low-precision arithmetic during the training process itself, allowing the model to adapt its weights to the quantized regime. PTQ is faster and simpler; QAT typically yields higher accuracy at very low bit-widths.

Modern quantization formats like GGUF encode not just the quantized weights but also the metadata needed to dequantize them during inference. Techniques such as GPTQ, AWQ, and the k-quant methods used by llama.cpp offer different trade-offs between compression ratio, speed, and quality. A well-quantized 7B model at 4-bit precision can fit into 4 GB of RAM and run on a laptop CPU — a stark contrast to the 14 GB required for the same model at FP16.

Why It Matters

Without quantization, running large language models requires expensive GPU hardware with substantial VRAM. A 70B-parameter model at FP16 needs roughly 140 GB of memory — far exceeding any single consumer GPU. Quantization democratizes access to powerful models by making them runnable on commodity hardware, edge devices, and even mobile phones. For organizations, this translates directly into lower infrastructure costs, reduced latency, and the ability to deploy AI in privacy-sensitive environments where data cannot leave the local device.

How It Works

The quantization pipeline starts by analyzing the distribution of weight values in each layer of the trained model. A mapping function is then computed that converts each floating-point weight to its nearest low-precision counterpart while minimizing the overall reconstruction error. For INT8 quantization, this typically involves computing a scale factor and zero-point per tensor or per channel. For aggressive 4-bit schemes, grouping strategies (e.g., quantizing in blocks of 32 or 128 weights) help preserve accuracy. The quantized model is then serialized in a deployment-ready format such as GGUF, which stores the quantized weights alongside the dequantization parameters needed at inference time.

Example Use Case

A healthcare startup needs to run a fine-tuned 13B medical Q&A model on hospital workstations that have no dedicated GPU. By quantizing the model from FP16 to Q4_K_M using llama.cpp's GGUF format, they reduce the model size from 26 GB to 7.4 GB. The quantized model runs at 12 tokens per second on CPU alone, with less than 1% degradation on their medical benchmark — enabling real-time clinical decision support without sending patient data to the cloud.

Key Takeaways

• Quantization reduces model precision (FP16 → INT8 → INT4) to shrink memory requirements by 2–4x or more.
• Post-training quantization is quick and easy; quantization-aware training yields better results at very low bit-widths.
• GGUF is the most popular format for distributing quantized models for local inference.
• Well-executed quantization preserves 95–99% of model quality while dramatically lowering hardware requirements.
• Quantization is essential for deploying LLMs on edge devices, laptops, and privacy-sensitive environments.