LLM API pricing explained: tokens, caching, batch, and hidden costs

A price sheet for an LLM API looks simple: two numbers per model. The bill you actually receive depends on a dozen smaller line items, and the cheapest model per token is often not the cheapest way to get a job done. This guide explains every part of the sheet, shows where costs hide, and gives you a fair way to compare providers.

What you are buying: tokens

LLM APIs bill by the token, the chunk of text a model reads and writes. In English a token averages about three-quarters of a word (see how many tokens are in 1,000 words). Prices are quoted per million tokens, because per-token prices are too small to read. A price of $3 per million tokens is $0.000003 per token, and a 1,000-word document of about 1,333 tokens costs around four-tenths of a cent to read at that rate.

Everything in this guide uses illustrative figures, not any provider’s actual prices. Real prices differ by provider and model, and they change often, so confirm current numbers on the official pricing page before you decide anything.

Anatomy of a price sheet

These are the line items you will meet, what each one means, and what to check:

Why output costs more than input

Input and output are priced differently because they use hardware differently. The model reads your whole prompt in a single parallel pass, which uses the GPU efficiently. It writes the answer one token at a time, and every step needs the weights and the growing cache to be read from memory again. Generation is slow and memory-bound, so a token of output ties up expensive hardware for longer than a token of input. Providers typically charge a few times more per output token, and our examples use a 5× ratio.

The practical consequences are direct. Asking for a short, structured answer is a real saving. A task that reads a lot and writes a little, such as classification or extraction, is cheap per task. A task that writes a lot from a small prompt, such as drafting long articles, is not.

Caching and batch: the two big discounts

Prompt caching

Most real applications resend the same long prefix on every call: instructions, policies, a reference document, a set of examples. Caching lets the provider reuse its earlier computation for that prefix and charge a reduced rate, often a small fraction of the normal input price. The details differ between providers. Some cache automatically, while others need you to mark the cacheable section. Entries expire after minutes of inactivity, and the prefix must match exactly, so putting a changing value such as a timestamp near the top of the prompt can defeat the cache. Design prompts with the stable content first and the variable content last.

In the worked example in the cost guide, caching a 1,200-token system prompt cut the cost per request by 22%.

Batch processing

If a job does not need an immediate response (nightly summaries, bulk classification, evaluation runs, data labeling), a batch interface can often cut the price roughly in half in exchange for a delivery window measured in hours. Because providers can schedule batch work when capacity is free, they pass some of the savings on. Plan for results arriving late, and for individual failures within a batch.

Where costs hide

• Reasoning tokens. Models that think before answering produce hidden tokens that are billed as output. A reply of 350 visible tokens might carry 1,500 more that you never see, multiplying the output bill by five. Look for a separate reasoning count in the usage data, and use effort or budget settings when offered.
• Context growth in chat. Each turn resends the history, so cost per turn rises through a conversation and the total grows roughly with the square of the turns.
• Agents and loops. An agent that takes ten steps pays for the accumulated context ten times. Cost per task can be an order of magnitude above cost per call.
• Retries. Rate-limit errors, timeouts, and validation failures that trigger regeneration each add another billed request.
• Tool definitions. A large set of function schemas rides along on every request as input tokens.
• Retrieval over-fetching. Sending twenty chunks when three would do multiplies input cost with no gain in quality.
• Tier and region surcharges. Priority processing, data residency options, and dedicated capacity can all carry premiums.
• Minimum commitments. Reserved or provisioned throughput is cheaper per token at high utilization and wasteful when idle.

Model tiers: a worked comparison

Providers typically offer a ladder of models: a small, fast tier; a balanced mid tier; and a large frontier tier. Price rises steeply with each step. The following table is a hypothetical ladder, applied to a request with 3,150 input tokens and 350 output tokens:

The spread between the small and frontier tier is more than 100× here. That is why model choice, not prompt tweaking, is usually the biggest single cost decision, and why many production systems route easy requests to a cheap model and send only hard ones to an expensive one.

Cheaper per token is not always cheaper per outcome

Token price is an input to your cost, not the cost itself. What matters is the cost of getting a correct, usable result, and a weaker model that fails more often shifts the cost to retries, review, or unhappy customers.

A hypothetical comparison for 1,000 support tickets. Suppose the small model resolves 60% without a human and the mid model resolves 95%, and each unresolved ticket costs $2 of human time:

The mid-tier model costs twenty times more in API fees and seven times less overall. The numbers are made up, but the shape is real: when failures are expensive, quality dominates token price. When failures are cheap (say, a draft that a person will rewrite anyway), the small model wins. Measure the success rate of each candidate on a test set drawn from your own data, then compute:

API or self-hosting?

Open-weight models can be run on your own or rented GPUs, which turns a per-token price into a per-hour hardware cost. The break-even depends almost entirely on how busy the hardware is.

Illustration: a GPU costs $2 per hour, and with batching it sustains 1,500 generated tokens per second across all users. That is 5.4 million tokens per hour, or about $0.37 per million output tokens at 100% utilization. At 20% utilization the same hardware yields about $1.85 per million tokens, because you pay for the idle hours. Real throughput varies widely with the model, the context length, the batch size, and the serving software, so measure it.

Self-hosting tends to make sense when volume is high and steady, when you need data to stay in your environment, or when you have a specialized fine-tuned model. It tends to lose when traffic is spiky or small, and it adds engineering work: serving software, monitoring, capacity planning, and upgrades. Sizing the hardware starts with the GPU VRAM guide.

How to compare providers fairly

1. Fix the workload. Take a sample of real requests with measured input and output tokens, as in the cost calculation guide.
2. Count with each provider’s tokenizer. The same text produces different token counts across providers, so a lower price per token is not always a lower price per request.
3. Run the same test set through each candidate and score the outputs against your quality bar.
4. Apply every discount and surcharge that really applies: cache hit rate, batch eligibility, long-context thresholds, reasoning tokens, tool fees.
5. Compute cost per successful outcome, not per token.
6. Check the non-price terms: rate limits, latency, uptime history, data retention and training policies, regional availability, and how much notice you get before models are retired.
7. Re-run the comparison on a schedule. Prices fall and new models arrive often, and a choice that was right six months ago may not be today.

FAQ