How to choose a GPU for LLMs: VRAM, bandwidth, and budget

Choose an LLM GPU in this order: calculate the peak memory requirement, reject cards that cannot fit it, then compare memory bandwidth, serving features, and total cost. This guide gives you a vendor-neutral worksheet so a stale price or a model name does not hide the real constraint.

How should you choose a GPU for an LLM?

First estimate weights + KV cache + runtime overhead for the peak model and workload. Select a card or multi-GPU setup with enough usable VRAM, not merely enough to load the weights. After capacity is satisfied, compare memory bandwidth, interconnects, software support, power, availability, and purchase or rental cost.

There is no reliable universal “best GPU.” The right choice depends on model precision, context length, concurrency, latency target, utilization, and whether you own the hardware or rent it. Prices and stock change, so this guide uses no current vendor price claims; check official product and provider pages before buying.

Why is VRAM the first filter?

VRAM is the GPU’s local memory. An inference workload uses it for model weights, the KV cache, activations, kernel workspaces, allocator overhead, and sometimes a draft model or vision encoder. The planning equation is:

Weights depend on parameter count and precision. Quantizing FP16 to INT8 or INT4 can reduce weight memory, but it does not automatically reduce the KV cache. Cache memory depends on architecture, tokens, cache precision, and concurrent sequences; calculate it separately with the KV-cache formula.

Use peak, not average, assumptions for a service with a latency or availability promise. If you normally serve 2,000-token prompts but must accept 16,000-token requests, decide whether to size for the peak or enforce a hard limit. A bigger card may be cheaper than repeated out-of-memory failures, but the only honest comparison uses your actual workload.

Worked fit check: an 8B-shaped serving workload

Suppose the weights require 16.1 GB, the peak KV cache requires 8.6 GB, and you budget 15% overhead. The raw subtotal is:

The 24 GB card fails the capacity check by about 4.41 GB. A 32 GB card has nominal capacity above the estimate, but usable capacity, runtime reservations, and the actual decimal-versus-binary unit should still be measured. If you quantize the weights, the requirement may fall; if you increase concurrency, context, or cache precision, it may rise.

The GPU fit calculator makes these assumptions explicit. It does not recommend a product because it cannot know your current workload, cooling, provider terms, or software stack.

Why does memory bandwidth matter after fit?

Capacity answers “can the model and workload reside on the card?” Bandwidth affects how quickly the runtime can move weights and cache data through the GPU. Autoregressive generation often reads a large amount of model state for each new token, so a card with higher usable bandwidth can deliver better generation throughput when memory movement is the bottleneck.

Bandwidth is not a simple speed guarantee. Kernel quality, quantization format, batch size, attention implementation, interconnect traffic, thermal limits, and the model itself all matter. Treat the manufacturer or provider bandwidth specification as a comparison input, not as a promised tokens-per-second benchmark.

A practical rule is to reject cards that fail VRAM, then prefer the option with more bandwidth when latency and throughput matter. That rule fails when the workload is compute-bound, when software support is much better on a slower card, or when the faster card’s capacity forces a multi-GPU arrangement with communication overhead.

How should budget enter the decision?

Separate capital cost from operating cost. Owning a card involves purchase price, electricity, cooling, depreciation, maintenance, and the cost of idle capacity. Renting replaces the purchase with an hourly or per-use rate, but availability, storage, egress, and minimum billing can matter. Provider terms and prices change, so collect current numbers from official pages for the exact region and configuration.

Convert options to a common unit such as cost per successful request, cost per generated million tokens, or monthly cost at a measured utilization. Include the quality and latency effects of quantization, retries, batching, and downtime. A cheap card that cannot meet the SLA is not cheap for the product.

For a low or spiky workload, renting can avoid paying for idle hardware. For steady high utilization and stable capacity, owning can become attractive. The break-even condition fails when utilization assumptions are wrong, hardware needs change quickly, or engineering time becomes the dominant cost. Measure usage before making a long commitment.

Should you use one GPU or several?

If one card has enough usable VRAM, a single-GPU deployment is usually simpler: fewer communication paths, simpler placement, and fewer failure points. When the model or peak cache exceeds one card, sharding can spread weights and cache across several GPUs. The arithmetic capacity is roughly additive, but each device still needs its own runtime buffers and the workload must communicate across the interconnect.

Tensor parallelism can improve fit and throughput when the GPUs have a fast interconnect, but communication overhead can erase the benefit on loosely connected cards. Pipeline parallelism changes the execution schedule and can introduce bubbles or uneven stage utilization. Treat multi-GPU as a serving-system design, not as simply adding the VRAM labels together.

A good rule is to test the smallest configuration that meets peak memory, latency, and concurrency requirements. It fails when redundancy, fault isolation, or future growth matters more than the minimum footprint; in those cases, deliberately leave capacity for a second replica or a larger model revision.

How do GPU choices go wrong?

The most common mistake is buying for weight memory alone. A quantized model can load successfully and then fail when the first long prompt creates a large KV cache. Calculate the full peak request and test concurrency.

Another mistake is comparing advertised VRAM labels without leaving operational headroom. A card’s total memory is not a promise that every byte is available to your model. The driver, runtime, display server, allocator, and other processes consume memory. Keep a reserve and inspect peak usage with the actual serving engine.

Do not use a bandwidth number as a substitute for an application benchmark. Two cards can have similar theoretical bandwidth and different performance because their kernels, interconnects, or supported formats differ. Benchmark the exact model, precision, context, batch, and server version.

Finally, do not optimize only for the purchase price. Power, noise, cooling, cloud utilization, model quality, and engineering effort can dominate total cost. Keep the model, runtime, and hardware replaceable until the workload is measured.

Next steps: run the worksheet

Use the GPU fit calculator for a first capacity and bandwidth comparison. Then read how to estimate GPU VRAM, quantization explained, and KV cache explained to replace rough inputs with model-specific measurements.

FAQ