How much VRAM does an LLM need?
Model weights take parameters × bytes per parameter. The KV cache and runtime overhead then add to that, and training needs several times more. This guide shows how to estimate each part.
The formula
weights ≈ parameter count × bytes per parameter
Bytes per parameter depend on precision: 2 for FP16/BF16, 1 for 8-bit and 0.5 for 4-bit. This guide uses GB (109 bytes). Tools that report GiB (230 bytes) will show a number about 7% smaller, so 14 GB is roughly 13 GiB.
Weight memory by model size
| Parameters | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 3B | 6 GB | 3 GB | 1.5 GB |
| 7B | 14 GB | 7 GB | 3.5 GB |
| 13B | 26 GB | 13 GB | 6.5 GB |
| 34B | 68 GB | 34 GB | 17 GB |
| 70B | 140 GB | 70 GB | 35 GB |
These are weights only. Real quantized files are usually a little larger because formats store scaling metadata, so treat the 8-bit and 4-bit columns as lower bounds.
The KV cache grows with context
While generating text, the model keeps the attention keys and values for every token so far. That is the KV cache, and it grows with context length and batch size:
Take an illustrative 7B-class architecture with 32 layers, 32 KV heads and a head dimension of 128, stored in 16-bit. Each token costs 2 × 32 × 32 × 128 × 2 = 524,288 bytes, about 0.5 MB. A 4,096-token context then needs roughly 2.1 GB, and 8,192 tokens roughly 4.3 GB. Models that use grouped-query attention have fewer KV heads, which shrinks this cache proportionally. Check your model's config before trusting a generic number.
Worked example: 7B at 4-bit with an 8K context
- Weights: 3.5 GB (add a little for quantization metadata)
- KV cache at 8,192 tokens, batch 1: about 4.3 GB
- Subtotal: about 7.8 GB
- Overhead at 10–20% (CUDA context, buffers, fragmentation): about 8.6–9.4 GB
An 8 GB card is too tight for this setup, and a 12 GB card is comfortable. Notice that the KV cache here is larger than the weights. That is why a "fits in 8 GB" claim based only on parameter count can be wrong once you use long contexts.
Training needs much more
Full fine-tuning with Adam in mixed precision is commonly estimated at about 16 bytes per parameter: weights, gradients, and optimizer states including an FP32 master copy. For a 7B model that is roughly 112 GB before activations. Parameter-efficient methods such as LoRA and QLoRA train only small adapter weights, which is why they can fit on far smaller GPUs. Treat any weights-only number as a floor for training, not a capacity guarantee.
Common mistakes
- Counting only the weights and ignoring the KV cache.
- Mixing up GB and GiB, or bits and bytes.
- Assuming a 4-bit model costs exactly one quarter of 16-bit in practice.
- Ignoring batch size, which multiplies the KV cache.
- Using inference numbers to plan a training run.
Frequently asked questions
Can I run a 70B model on a 24 GB GPU?
Not fully on the GPU. Even at 4-bit the weights alone are about 35 GB. Some runtimes can offload layers to system RAM, but generation gets much slower.
Is system RAM a substitute for VRAM?
It can hold overflow, but memory bandwidth is far lower, so speed drops sharply. Plan for the model to fit in VRAM.
How accurate are these estimates?
They are planning figures. Actual usage depends on the runtime, the model architecture, and your settings, so measure peak memory on your own workload before buying hardware.
Run the numbers
Use the GPU VRAM Estimator with an overhead factor that matches your setup. If you are also budgeting API spend, read how to calculate LLM API cost and how many tokens are in 1,000 words. For evaluating a model once it runs, see precision vs recall vs F1.