Model quantization is one of the most frequently mentioned techniques in LLM deployment. Its goal is simple: cut VRAM usage and compute cost with as little quality loss as possible, so one GPU can host a larger model or serve more concurrent requests.
1. Why quantization matters
Take a 7B model. Stored in FP16 (2 bytes per parameter), the weights alone take roughly 14 GB. Add activations, KV cache and framework overhead, and the real footprint often exceeds 16 GB — too much for a mainstream consumer GPU. Compressed to 4-bit (INT4), the weight size drops to roughly a quarter, so a 7B model needs only about 3.5–4 GB and can run on a single consumer card. These are theoretical estimates; actual usage varies with the quantization scheme, context length and batch size.
2. The basic idea
Quantization maps continuous floating-point values onto a limited set of discrete integers. The simplest symmetric form is q = round(x / s), with de-quantization x ≈ s × q, where s is the scale. In practice a zero-point is added for asymmetric quantization, and group-wise schemes compute a separate scale for each small block of weights — smaller groups mean higher precision but more metadata overhead.
3. Common approaches
By timing: Post-Training Quantization (PTQ) converts a finished model directly and is the cheapest, most common route; Quantization-Aware Training (QAT) simulates quantization error during training or fine-tuning and usually gives better low-bit accuracy at the cost of retraining. By bit width: INT8 is nearly lossless, INT4 is today's mainstream compromise for local deployment, and 3-bit or lower usually needs more sophisticated methods. Tooling-wise, common routes include GPTQ, AWQ, bitsandbytes NF4 and the GGUF formats used by llama.cpp, each trading off GPU versus CPU inference differently.
4. What you lose
The lower the bit width, the more noticeable the degradation on long-chain reasoning, math and code tasks; chat and summarization tolerate quantization better. Evaluate a quantized model on your target workload instead of relying on perplexity alone.
5. When it is worth it
If you want to run locally, cut inference cost or serve more concurrency per GPU, quantization is close to mandatory. If you chase maximum quality and have plenty of compute, staying on FP16/BF16 is still the safer choice.
Reference: compiled from publicly released industry information (documentation of GPTQ, AWQ, bitsandbytes, llama.cpp and similar projects).