KV cache explained: the memory formula for LLM inference
The KV cache stores attention keys and values from tokens the model has already processed, so generation does not recompute the whole conversation at every step. Its memory grows with layers, KV heads, context length, cache precision, and concurrent sequences—often becoming the largest part of a real serving workload.
What is a KV cache?
The key-value cache, usually shortened to KV cache, is inference memory that holds the attention keys and values for tokens already present in an active sequence. During autoregressive generation, the model can reuse those tensors instead of recalculating them for every new output token.
The important planning consequence is simple: weights scale with model size, but KV-cache memory scales with the amount of active text. Longer contexts and more simultaneous sequences require more cache even when the model weights do not change.
What does the KV cache store?
In a transformer attention layer, each token is projected into query, key, and value vectors. The query is used for the current step; the keys and values for earlier tokens are retained. Each layer has its own cache. For a decoder-only model, the cache stores both a K tensor and a V tensor, which is why the formula starts with a factor of 2.
The number of KV heads is not always the same as the number of query heads. Traditional multi-head attention can use one key-value head per query head. Grouped-query attention shares one KV head across several query heads, reducing cache memory while keeping more query heads for computation. Read the model configuration rather than assuming the counts are equal.
Cache precision is another input. BF16 and FP16 use 2 bytes per element; FP8 or an 8-bit cache uses 1 byte in the simple estimate. Actual runtime layouts can add padding or use a different representation, so the formula is a planning model and peak-memory measurement is the final check.
What is the KV-cache memory formula?
Layers is the number of transformer layers. KV heads is the number of key-value heads. Head dimension is usually hidden size divided by the number of query heads, unless the model configuration gives it directly. Bytes per element is 2 for a 16-bit cache and about 1 for an 8-bit cache. Tokens per sequence includes the prompt and the generated tokens currently retained. Concurrent sequences is the number of active requests whose caches coexist.
Keep the unit conversion visible:
Use decimal GB when comparing with a decimal memory estimate and GiB when comparing with a tool that reports binary units. The numerical difference is about 7%, which matters when a card is nearly full.
Worked example: an 8B-shaped model at 8,192 tokens
Consider a model shaped like an 8B grouped-query-attention model: 32 layers, 8 KV heads, and a head dimension of 128. Assume a BF16 cache, so each cached element uses 2 bytes. We first calculate one token for one sequence.
| Step | Arithmetic | Result |
|---|---|---|
| Keys and values | 2 | Both tensors |
| Bytes per token | 2 × 32 × 8 × 128 × 2 | 131,072 bytes |
| KiB per token | 131,072 ÷ 1,024 | 128 KiB |
| One sequence at 8,192 tokens | 131,072 × 8,192 | 1,073,741,824 bytes |
| Decimal size | 1,073,741,824 ÷ 1,000,000,000 | 1.07 GB |
| Binary size | 1,073,741,824 ÷ 1,073,741,824 | 1.00 GiB |
Eight simultaneous sequences multiply the same one-sequence result by eight: 1,073,741,824 × 8 = 8,589,934,592 bytes, or 8.59 GB and 8.00 GiB. The KV-cache calculator lets you change each architectural and workload assumption.
This is cache memory only. If the model’s weights use 16.1 GB and the cache uses 8.59 GB, the subtotal is 24.69 GB before runtime overhead. That is why the full VRAM estimate must add weights, cache, and buffers together.
What makes KV-cache memory grow?
Context length. Doubling tokens doubles the cache for one sequence. A 32,768-token sequence with the example shape needs four times the cache of an 8,192-token sequence: 4.29 GB rather than 1.07 GB.
Concurrency. Eight active sequences need eight times the cache of one sequence if they all hold the same number of tokens. A server that promises high concurrency must budget peak active tokens, not just the average prompt.
Architecture. More layers, more KV heads, and a larger head dimension all increase cache linearly. Grouped-query attention can reduce KV heads. Multi-query attention can reduce them further, but architecture alone does not tell you the exact number; inspect the configuration.
Cache precision. Moving from a 16-bit cache to an 8-bit cache roughly halves the arithmetic estimate. It can introduce quality or compatibility trade-offs, and not every runtime supports every cache format. Weight quantization and KV-cache quantization are separate decisions.
Allocation policy. Some serving engines allocate cache blocks as tokens arrive, while others reserve more capacity up front. Paged allocation can avoid reserving the full maximum for every request, but it does not make peak active tokens free. Sliding-window attention or eviction can bound memory for models that support those patterns.
How does KV-cache planning go wrong?
The most common mistake is calculating one sequence and deploying many. A 1.07 GB cache may look harmless beside the weights, but eight users turn it into 8.59 GB before overhead. Write the concurrency assumption next to the memory result and size for the peak you intend to serve.
Another mistake is using query heads in place of KV heads for a grouped-query model. If a model has 32 query heads but 8 KV heads, using 32 in the formula overestimates the cache by 4×. That conservative mistake can waste hardware; using 8 when the model actually has 32 underestimates it by 4× and can cause an out-of-memory failure. Read num_key_value_heads or the equivalent configuration field.
Do not confuse maximum context with current cached tokens. A model may advertise a large maximum, but a paged runtime can allocate memory closer to the tokens currently active. Conversely, a setting that reserves the maximum sequence length can make the high-water mark appear immediately. Observe peak allocation under realistic prompts, output lengths, and request overlap.
Finally, do not assume quantizing weights fixes a cache bottleneck. A 4-bit model can have only a few gigabytes of weights and still need more memory for a long BF16 cache. Reduce maximum tokens, cap concurrency, use a supported lower-precision cache, or choose an architecture with fewer KV heads when the cache—not the weights—is the limiting term.
Next steps: size the complete workload
Try the KV-cache calculator with the 32-layer, 8-head, 128-dimension example. Then combine its output with the GPU VRAM estimator and the quantization guide. If you are comparing self-hosting with an API, connect the memory requirement to LLM API pricing.
FAQ
What does KV cache stand for?
KV cache stands for key-value cache. It stores the attention keys and values for tokens already processed so autoregressive generation can reuse them instead of recomputing the full history.
How much memory does KV cache use per token?
KV bytes per token equals 2 × layers × KV heads × head dimension × bytes per cache element. The exact result depends on the model architecture and cache precision.
Does KV cache grow with model parameters?
Not directly with total parameter count. It grows with attention layers, KV heads, head dimension, cache precision, tokens per sequence, and concurrent sequences, so two models with similar parameter counts can have different cache requirements.
Does quantization reduce the KV cache?
Weight quantization does not automatically reduce the KV cache. The cache changes only if the runtime separately stores keys and values at a lower precision or uses another memory-saving attention strategy.
How can I reduce KV-cache memory?
Reduce active tokens or concurrency, use a model with fewer KV heads, choose a supported lower-precision cache, use paged allocation, or apply a bounded attention strategy when the model and task allow it.