Learn / What is model quantization?
What is model quantization?
Model quantization reduces the numerical precision of a model's weights and activations — for example from 16-bit to 8-bit or 4-bit — so the model uses less memory and runs faster. Done well, it preserves most of the model's accuracy while cutting cost and latency.
How it works
A model's parameters are normally stored as 16-bit numbers. Quantization maps those values to a smaller numeric type using fewer bits, shrinking the model's memory footprint and reducing the data the GPU moves and computes per token. Less memory traffic and smaller math generally mean faster inference and lower cost.
Common precisions
| Precision | Bits | Typical use |
|---|---|---|
| FP16 / BF16 | 16 | High-quality baseline |
| INT8 | 8 | Strong size and speed trade-off |
| INT4 | 4 | Maximum compression |
Benefits and trade-offs
Quantization makes models smaller, faster, and cheaper to serve, and can let a model fit on fewer or smaller GPUs. The trade-off is potential accuracy loss, which grows at lower precisions. The right choice balances quality against cost and latency for your specific task.
How Netra helps
Netra Runtime is a high-performance inference platform. In its published benchmark, the same compressed weights were loaded into every engine on a single NVIDIA H200 GPU, and Netra served up to 4.4x more output per second than vLLM — showing that the serving runtime, not just the precision of the weights, drives production performance. See the Netra vs vLLM comparison or how to reduce LLM inference cost.
Frequently asked questions
Why quantize a model?
Quantization shrinks a model so it uses less GPU memory and runs faster, which lowers cost and latency and can let a model fit on smaller or fewer GPUs.
Does quantization reduce accuracy?
It can, but modern methods usually keep accuracy loss small, especially at 8-bit. Lower precisions like 4-bit save more memory but need more care to preserve quality.
What precisions are commonly used?
Common choices are 16-bit (FP16 or BF16) as a high-quality baseline, 8-bit (INT8) for a strong size and speed trade-off, and 4-bit (INT4) for maximum compression.