Quantization explained: FP16 vs INT8 vs INT4 for LLMs

Quantization stores model weights with fewer bits, reducing memory and often making local inference possible on smaller GPUs. This guide explains what changes numerically, works through a 7B-model example, and separates weight savings from the cache, activations, and quality trade-offs that remain.

What is quantization?

Quantization is the process of representing neural-network numbers with a smaller numeric format. For LLM inference, the main target is usually the model’s weights—the learned parameters that must be loaded into memory. Moving from 16 bits per parameter to 8 or 4 reduces weight storage roughly in proportion to the bit count.

The simplest estimate is weight memory = parameter count × bits per parameter ÷ 8. It is a weights-only estimate. It does not automatically halve the KV cache, activations, temporary buffers, or the memory used by other models loaded in the process.

How does LLM quantization work?

A quantizer maps a range of higher-precision values to a smaller set of representable values. A group of weights may share a scale and sometimes a zero point; the stored low-bit code is decoded back to an approximate value when the layer runs. The approximation introduces quantization error, meaning the decoded value is not exactly the original value.

Quantization can happen after training, which is called post-training quantization, or it can be considered during training with quantization-aware methods. Post-training workflows are convenient because they start from an existing checkpoint. They still need careful calibration or evaluation because some layers and tasks are more sensitive than others.

The raw formula assumes every parameter uses exactly the target bit width. Real files also store scales, zero points, metadata, alignment padding, and sometimes unquantized layers. For that reason, an INT4 file often behaves like slightly more than 4 bits per parameter. The right number is the file size or the quantization format’s documented accounting, not the label alone.

FP16 vs INT8 vs INT4: what is the difference?

FP16 and BF16 are floating-point formats, not integer quantization in the strict sense. They preserve a relatively wide numeric range and are common for inference and fine-tuning. They are a useful reference because a 7-billion-parameter model needs about 14 decimal GB for weights at 16 bits.

INT8 stores integer-like values with a scale that maps them back to the model’s range. It usually cuts raw weight memory in half compared with FP16. It can be a good default when you want a smaller model but want to keep evaluation risk modest.

INT4 compresses each weight to four nominal bits and typically requires extra metadata. It can reduce a 7B model’s raw weight estimate from 14 GB to about 3.5 GB before metadata, but the actual file may be closer to 4 GB depending on the method and which layers stay at higher precision. The quality impact depends on the model, calibration data, task, and implementation.

Do not treat format names as interchangeable across tools. “4-bit” can describe different group sizes, scales, activation handling, and kernels. A file that is small on disk may still be loaded into a different working format by a runtime, and an accelerator may have optimized support for one format but not another.

Worked example: a 7B model at 16-bit and 4-bit

Assume a model has exactly 7 billion parameters. We compare FP16 weights with an INT4 representation that has 12.5% metadata overhead. The overhead is an illustrative planning assumption; the actual effective size depends on the quantization format.

The quantization calculator reproduces this example. The values are decimal GB because the arithmetic uses 10⁹ bytes per billion parameters. A monitoring tool may report GiB, where 14.00 GB is about 13.04 GiB; do not compare the two units without converting.

This example only answers “can the weights fit?” A serving decision must add the KV cache and runtime overhead. If the model uses a long context or several concurrent sequences, the cache can be larger than the quantized weights.

What does quantization change besides memory?

Quality. Lower precision throws away information. Many models remain useful at 8-bit and 4-bit, but the loss is not a universal constant. Tasks with precise arithmetic, code generation, small factual distinctions, or long-context retrieval may be more sensitive. A practical rule is to start with 8-bit when quality risk is expensive, then test 4-bit on a representative evaluation set. The rule fails when the model cannot fit at 8-bit; in that case, 4-bit may be the only viable experiment.

Speed. Smaller weights can reduce memory traffic and improve throughput, but only if the runtime has an efficient kernel for the format. Dequantizing values during computation can add work. A format that saves memory but lacks optimized kernels may be slower than a larger format. Measure tokens per second and latency on the target hardware.

Cache and activations. Weight quantization does not automatically quantize the KV cache. Many runtimes keep the cache in FP16 or BF16 for quality, while some support FP8 or other cache formats. The cache is driven by layers, KV heads, head dimension, tokens, and concurrent sequences; use the VRAM estimation guide for that calculation.

Fine-tuning. A quantized frozen base can support parameter-efficient methods such as QLoRA, where adapters are trained while the base weights remain quantized. That does not mean full training fits in the same memory as inference: gradients, optimizer states, activations, and temporary buffers still need room.

How does quantization go wrong?

The first mistake is subtracting the saved weight memory from the entire VRAM requirement. A 7B model that drops from 14 GB to about 4 GB does not free exactly 10 GB if its KV cache already uses 8 GB and the runtime needs additional buffers. Quantization helps the weights; it does not erase the rest of the workload.

The second mistake is choosing a format from its label without checking the file and runtime. Group scales, unquantized embeddings, alignment, and loader behavior can move the effective bits per parameter. Inspect the artifact size, measure peak memory after loading, and confirm which cache precision is active.

The third mistake is evaluating only a few easy prompts. A quantized model can look fine in casual chat and fail on the task that matters: a schema-constrained extraction, a coding benchmark, or a domain-specific classification set. Keep a small regression test set, compare the candidate against the higher-precision baseline, and inspect failure types rather than relying on one average score.

Finally, “smallest file” is not always the right objective. If INT4 causes retries, human review, or lower answer acceptance, the operational cost can exceed the memory saved. Choose the lowest precision that meets your quality and latency requirements, not the lowest number printed on the filename.

Next steps: test the fit, then test the quality

Use the quantization calculator to compare source and target formats, then use the GPU VRAM estimator for a first weights-and-overhead check. For the part quantization does not cover, read how to estimate GPU VRAM for LLM inference. If your workload is API-based rather than self-hosted, compare its token economics in LLM API pricing explained.

FAQ