Learn / What is prompt caching?

What is prompt caching?

Prompt caching stores the computed state of a prompt's beginning so that the next request starting with the same tokens can skip recomputing it. Because chat histories, system prompts, and tool definitions repeat verbatim across requests, the savings are large: cached input is typically billed at around a tenth of the normal rate, and time-to-first-token drops because the engine only processes what changed.

Why prompts repeat

The API is stateless: the model remembers nothing between calls, so a chat client resends the entire history on every turn. That makes billed input grow with the square of conversation length — in a worked example from our token cost deep dive, twenty turns of a modest conversation bill 117,200 input tokens while the user actually typed 2,400. The expensive part of every request is a prefix the engine has already seen, which is exactly the pattern caching exists for.

What it's worth

If a fraction $f$ of your input hits the cache and cached tokens cost a ratio $r$ of the normal price, your effective input price becomes:

p_{eff} = p_{in} ((1 - f) + f \cdot r)

with $r \approx 0.1$ at typical provider pricing. In a long conversation $f$ tends towards 1, which turns a 90% discount on the dominant cost term into the single most valuable change you can make to a chat or agent workload. Latency improves for the same reason: prefill work for the cached prefix simply never runs.

How to structure prompts for cache hits

Stable material first. Keep the system prompt, tool definitions, and retrieved documents at the front of the prompt, and the variable material (the user's latest message) at the end.
Never inject volatility near the top. A timestamp, request id, or randomized example early in the prompt invalidates everything after it, because caching works on an exact shared prefix.
Keep history append-only. Summarizing or reordering old turns rewrites the prefix and forfeits the hit; if you truncate, truncate from a stable boundary.

How it works under the hood

What actually gets cached is KV state: the key and value vectors the model computes for every prompt token during prefill (see what is a KV cache?). In engines built on paged attention, KV memory lives in fixed-size blocks addressed through a table, so two requests sharing a prefix can point at the same physical blocks with a reference count. Block hashes decide when reuse is safe. Serving-side prefix caching and provider-side billed discounts are the same mechanism seen from two sides: the provider passes part of the saved compute back as a lower price.

Frequently asked questions

Does prompt caching change the model's output?

No. The cached KV state is identical to what the model would recompute from the same tokens, so generation proceeds exactly as if the whole prompt had been processed fresh. Caching changes cost and latency, not results.

How much does prompt caching save?

Cached input is typically billed at around a tenth of the normal input rate. In a long conversation the cache-hit fraction approaches 1, so the dominant cost term earns roughly a 90% discount. Time-to-first-token also drops because prefill skips the cached prefix.

What invalidates a prompt cache?

Any changed token invalidates everything after it, because caching works on an exact shared prefix. Reordering sections, injecting a timestamp near the top, or editing the system prompt all break the cache for the material below the change.