LLM inference memory
KV cache calculator
Estimate cache memory per token and total usage from the model’s attention shape, cache precision, context length, and concurrent sequences.
Free · no sign-in · updates instantly
1. Enter the architecture and workload
Read from the model configuration
Key-value heads, not query heads
Usually hidden size ÷ query heads
The cache precision assumption
Prompt plus retained generated tokens
Active requests sharing the GPU
Try an example:
Per token0 bytes
Keys + values for one sequence
Total cache0 GB
Decimal memory estimate
Binary total0 GiB
For GiB-based monitoring tools
Limit: this estimates KV tensors only. Add model weights, activations, allocator overhead, and other runtime buffers when sizing a GPU.
How the KV-cache estimate works
KV bytes per token = 2 × layers × KV heads × head dimension × bytes per element
total bytes = KV bytes per token × tokens × concurrent sequences
The factor of 2 accounts for keys and values. A grouped-query model can have fewer KV heads than query heads, so use the model’s configuration. The result is a planning estimate; runtime layouts and allocation policies can change peak memory.
Read the full explanation
See KV cache explained for the worked arithmetic, concurrency trade-offs, and failure modes. For the complete workload, read how to estimate GPU VRAM.