Prompt Caching as a Cost Lever

A long prompt costs money every turn. The customer notices. The CFO asks about it on week three. Prompt caching is the cheapest lever you have in that conversation, and understanding what it actually does is what keeps it from backfiring into an incident.

What is cached

The name is misleading on first contact. Prompt caching does not cache responses; it caches the provider's internal processing of a prefix of the input. When the next request starts with the same bytes, the provider skips the work it already did and picks up where the prefix ends.

Anthropic, OpenAI and Google all expose a form of this, and the shape is the same across them. The cached prefix gets a steep discount — Anthropic's current numbers put a cache read at roughly a tenth of a cache write, which is itself at a slight premium over an ordinary input token. The hot path on a well-cached system runs on that tenth-price.

The honest picture of a cached request:

1. Step 1

   Request arrives

   The provider hashes a prefix of the input and looks it up.

2. Step 2

   Prefix hit

   The internal state from last time is restored. Nothing in the prefix is re-tokenised or re-attended-to.

3. Step 3

   Suffix is processed

   The new bytes after the prefix are processed normally and billed at the usual rate.

4. Step 4

   Response

   Output tokens are billed at the usual output rate — caching does not change output pricing.

Where the cached prefix should end

The design choice is where to draw the line between "cached forever" and "changes per request." The usual anatomy of an FDE's build:

[ system prompt ]          <-- cached
[ customer-specific policy ]  <-- cached per customer, warmed on first call
[ retrieved context ]       <-- changes per query, not cached
[ user message ]           <-- changes per turn, not cached

The system prompt and the customer policy can be stable across a whole tenant's day, so they go first. Retrieved context changes per query, so it goes after the cacheable region. The user message is always last.

The failure mode is putting something unstable early. A timestamp in the system prompt. A conversation id. A reference to "today's date." Every one of those is a byte change that invalidates everything downstream of it, and the cache serves nothing. The rule is simple: if it can move later, move it later.

Cache keys and the tenant wall

The second case is the incident most feared on a production AI deployment, and it is engineered by an FDE trying to improve a metric. The provider does not know your tenants. It caches what it is given. If you give it the same prefix for two tenants, you have told it they are the same session.

The discipline is one sentence: the prefix differs if and only if the content it should see differs. Everything else — the hit rate, the dollars, the latency chart — follows from that invariant. Break it, and no dashboard will catch the leak because the leak does not have an error signal.

The invisible miss an eval catches

The subtler cost of caching is the miss you did not notice. A cached system prompt that is six months old still produces responses, and those responses look fine. Then the product owner updates the policy document, and you discover that the deployment has been answering from the old one for a week — because the prompt caching the system prompt used was warm, and the update never made it into the prefix.

This is why caching and evals belong in the same conversation. Offline evals run against the current prompt, cached or not, so they catch a drift between "what we deployed" and "what is actually in the cache." Online evals catch the other direction: a cached prefix that was fine yesterday producing a response that trips a judge today. Neither eval has to be clever about caching — they just have to be run.

Talking to the CFO

The FDE's job in the cost conversation is not to defend the model choice. It is to show the lever. A modest but typical before/after on a well-designed deployment:

• A 12k-token system prompt, cached. 92% hit rate on the hot path.
• Average input tokens per turn drops from 12.8k to 2.4k (the suffix plus a few retrieved passages).
• Average per-turn cost on the hot path falls to roughly a third of the pre-cache number, before any model change.

The CFO cares about the third number. The FDE should walk in with it, not with a tutorial on how caching works. The engineering discussion is for the engineering meeting; the cost meeting wants the lever and the chart.

What to take away

Prompt caching stores the provider's processing of a stable prefix, so a later request that starts with the same bytes skips the work and pays a fraction for the prefix. Put stable things early and unstable things late; do not strand a cache with a timestamp in the system prompt. The prefix differs if and only if the content it should see differs — breaking that invariant is how a cost optimisation becomes a tenant-isolation incident. And the invisible miss is a stale cached prefix, which is why offline and online evals, run against whatever is deployed, are the thing that catches it. The CFO conversation is one number: average per-turn cost on the hot path, before and after.

Next: the start of Part Four, where we leave the prompt behind and look at retrieval — the layer that keeps the prompt small without lying about what the model knows.