Also known as: prompt cache, prefix caching API, cached prompts
Prompt caching is the API-side feature that lets you reuse a model provider's KV cache for a stable prompt prefix across requests. You mark the prefix as cacheable, the provider keeps its KV cache warm.
Prompt caching is how Anthropic, OpenAI, and Google expose the underlying KV cache to API users. You mark a prefix of your prompt as cacheable; the provider keeps its computed KV state warm; subsequent requests that share that prefix skip prefill on the cached tokens and pay roughly 10% of the normal input-token rate. Mechanically it is just prefix caching with billing semantics attached. Strategically it is the single highest-leverage cost optimization for RAG, agents, and long-system-prompt applications.
The mental model is identical across vendors; the surface differs.
cache_control: { type: "ephemeral" } on a content block. Up to 4 cache breakpoints per request. 5-minute TTL default, 1-hour option at 2x write cost. Cached reads at 10% of base input price.CachedContent objects you create, parameterize with TTL, and reference by handle. More work to wire up, more control over lifetime.The economics are similar in shape: a cache write costs slightly more than a normal input token, a cache read costs a fraction of one, and the break-even is usually two to three reuses of the same prefix.
Cache hits require exact prefix match, position by position. The discipline is to put everything that is stable up front and everything that varies at the end.
The recipe most production teams converge on:
Anthropic gives you four cache breakpoints; one common pattern is to place them at the boundaries above (system, tools, retrieved-context, history) so each layer caches independently. A new retrieval invalidates from layer 3 onward but leaves layers 1-2 hot.
A small change in any earlier layer cascades — a single appended sentence to the system prompt invalidates every cache below it. Treat the system prompt as a versioned artifact, not a string you tweak casually.
Prompt caching is not free; it is cheaper. Specifically:
Prompt caching wins biggest in three workloads. RAG with stable system prompts and rapidly varying user queries — the system prompt and tool block stay hot, and only the per-query retrieved passages and user message pay full price. Agent loops where the conversation grows append-only across many turns — every prior turn is cached prefix on the next iteration. And evaluation harnesses where you replay the same prompt across many candidate outputs — cache hit rates approach 100%.
Outside those, the savings are modest. For one-off, high-variance prompts (a typical chatbot’s first turn), caching is a small win at best. The discipline is to design the prompt for caching — pick orderings that maximize prefix reuse — rather than retrofit caching onto whatever shape the prompt happened to take.
Prompt caching is the cheapest 5-10x cost cut available to a RAG or agent stack. The price is treating your system prompt as a versioned artifact, not a string you tweak.
Anthropic offers a 5-minute default TTL with a 1-hour option at higher cost. OpenAI keeps prompts cached for 5-10 minutes idle, longer under sustained traffic. Google's Vertex AI uses explicit cache objects with developer-controlled TTLs. None are guaranteed durable — under memory pressure, the provider may evict early.
The KV cache is computed sequentially. A cache hit requires the new request's tokens to match the cached prefix exactly, position by position. Move one token in the prefix and the cache invalidates from that point onward. Stable content goes first; volatile content (the user message) goes last.
When prefixes change every request (no reuse possible), when total prefix size is below the provider's minimum (Anthropic requires 1024 tokens for Sonnet/Opus), or when traffic is too sparse to keep the cache warm. The cache write itself costs more than a normal input token, so a single-use cache is a net loss.
