Prompt caching is a prefix match, not a flag

Prompt caching lets a model reuse processing from a shared prompt prefix, reducing input cost and latency on a later request. The cost examples below use a cache-read rate of one tenth of ordinary input pricing; rates and minimum sizes depend on the model. (Tokens are the chunks of text a model reads and is billed by, roughly a few characters each.)

Caching does not make the context window larger. Cached tokens still count toward it. It makes repeated context cheaper to process. To know whether it helps a particular call, inspect the API's cache-read and cache-creation token counts; a configured breakpoint alone does not establish reuse.

I applied it across four parts of my own system in June 2026. Two needed work, one was already handled for me, and one was not my decision to make. The token counts below are from those runs. This post did not record the exact model ID, so they establish reuse in that experiment rather than a model-specific setup recipe.

The whole idea in one picture#

Every request you send a model is one block of text, and normally the model reads all of it, and charges you for all of it, every time. Caching changes that. You pick a point in the request, called a breakpoint. Everything from the start of the request up to that point is stored after the first call and reused on the next one, at about a tenth of the price. Everything after that point is read fresh every time, at full price.

The one constraint shapes everything else: the stored part has to begin at the very start of the request. You cannot cache a passage from the middle or the end, only a run that starts at the first character and stops at your breakpoint. That run is called the prefix, and reuse only happens when it comes back identical down to the byte. So the whole game is to arrange the request with the text that stays the same up front and the text that changes after it.

The four parts of my system below are four versions of that same problem, getting the unchanging text up front so the stored run actually repeats, and they did not all have the same answer.

The cache has three rules#

The prefix has to meet three conditions. A miss need not raise an error, so each one needs a check in the usage counters.

The rule	What it means	Break it and
It requires an exact prefix match	changing a block changes the prefix ending there and every longer prefix	an earlier prefix already stored at another breakpoint may still be reused; inspect the cache-read counters to see how much matched
The prefix has to be big enough	the minimum is specific to the model version and service; check the current documentation	a breakpoint on a shorter prompt is ignored, and no error is raised
The prefix has to come back soon	the five-minute setting used in this example charges a 25% write premium; a cache read refreshes the lifetime, and a one-hour setting has a different write price	a prefix you store but never reuse in time costs more than not caching at all

The current Anthropic cache table, checked on November 3, 2026, lists thresholds and read-price exceptions by model version. The earlier version of this post incorrectly gave one threshold for all Sonnet models and another for all Opus models. Use the exact model ID and usage counters when checking your own calls.

Layer one: the agent, already handled#

The first layer is Claude Code, the coding agent I work in all day. It already caches its own system prompt and tool definitions, and it trims the conversation as it grows, so there was nothing for me to build.

The one part I control is its memory. Claude Code loads a Markdown file at the start of each session where I keep durable facts about my projects, so I do not re-explain them every time. Mine is an index file that points to a set of small single-fact notes. Because that file is part of the prefix the agent already caches, the right move was not to cache it again but to keep it short: a bloated memory file just makes every cached prefix larger for no benefit. The lesson at this layer is to check whether the platform already caches for you before you do anything yourself. Here, it did.

Layer two: my own backend calls, the real work#

The second layer is my backend calling the model directly. It has several call sites: a chat assistant that answers questions about a finished portfolio review, a widget that answers questions about a blog post, and a few one-shot text generators (single request in, single response out, no back-and-forth). I checked each one against the three rules.

The chat assistant was the useful candidate: it resends the entire prior review on every turn. That review is large, identical from turn to turn, and reused many times. The one-shot generators had only a few hundred instruction tokens and little opportunity for reuse. Their cache eligibility would need checking against the exact model's minimum; short instructions alone do not rule it out across all models.

I turned caching on per call site rather than globally in the shared client, because of the third rule. A call site that is over the size floor but never reuses its prefix would pay the 25% storage premium and never earn it back. Caching everything by default would tax every one-shot call to benefit the few that repeat.

The cached prefix contains the review encoded as JSON, and the database that stores the review does not guarantee a consistent field order when it is read back. Encoded without a fixed order, the JSON came out as a slightly different sequence of characters on every turn, which under the first rule is a different prefix every time. The cache would have matched nothing while the code looked entirely correct. The fix was to sort the JSON fields into a fixed order so the bytes are identical on every turn.

On the same call site I added a separate control for conversation length. The assistant keeps only the last 25 turns of history. That was my chosen retention policy, not a demonstrated limit on what a useful conversation needs: older details can be lost, and a turn-count cap does not bound tokens if individual turns are large. Context size needs its own measurement even when the prefix is cached.

Measured against a typical review, the cached prefix is 5,231 tokens. On the first turn the model stores those tokens. On the second turn it reuses all 5,231 and charges for 93 new input tokens instead of about 5,300. At a one-tenth cache-read rate, that makes this follow-up's input cost about 12% of the uncached input cost. Output tokens are billed separately.

Layer three: retrieval, a reason not to build a vector store#

The third layer answers plain-language questions over the archive of past reviews. It loads the relevant reviews, trims them, and sends the whole set to the model. The trimmed archive is the large, stable part, so it is what should be cached. The problem was ordering: the code put the question first and the archive second, and caching can only store a run that begins at the very start of the request, so an archive sitting at the end could not be stored at all. Once the archive moved to the front, a second question over the same archive reused 2,541 stored tokens instead of paying for them again.

My archives still fit inside the model's context window, so sending the archive and caching it was simpler for this project than maintaining a vector database. I did not compare answer accuracy or end-to-end latency against a retrieval pipeline. The next step is to watch input size and answer quality: try a keyword filter if useful material becomes hard to find, and evaluate vector retrieval when the corpus or the questions justify it. Fitting inside the window is a capacity check, not proof that the model uses every passage well.

Layer four: a data server, not my decision#

The fourth layer is a private server that sends my own portfolio data to a separate model on request, over the Model Context Protocol (MCP, the open standard for exposing tools and data to a model). At this server-to-model boundary, the receiving side decides whether to cache. The server controls response size: cap the field that could grow without limit, stop sending duplicate copies of large results, and log each response's byte size. Those changes reduce context consumption. Caching separately reduces the cost of processing repeated context.

What it saved#

Surface	Cached prefix	New input charged, turn two	Input it would have charged without caching
Finance chat	5,231 tokens	93 tokens	~5,300 tokens
Corpus retrieval	2,541 tokens	a few dozen	~2,600 tokens

The finance row in plain terms: at the example's one-tenth cache-read rate, the follow-up costs the equivalent of 93 + 523.1 ordinary input tokens, compared with 5,324 without caching. That is about 88% less input cost for this observed hit. The counters show that both prefixes qualified for caching in these runs; they do not establish a threshold for other model versions.

Two caveats. First, volume. This is a low-traffic system, so the actual dollars saved are small, and I am not going to dress a small personal project up as a serious cost cut. The savings are real and would scale with traffic, but the traffic is not there yet.

Second, and more important: I measured those reused prefixes with no delay between turns, one request sent the instant the previous one returned. A stored prefix lasts five minutes. In real use a person reads the answer, thinks, and then types the next question, and if that gap runs past five minutes the stored prefix is already gone and the next turn pays full price again. So my measurement proves the mechanism works, not that it works under real human timing. The token counter that will answer that is now running in the production logs. If the reuse is not happening, the fix is a longer storage window, which is a setting, not a redesign. I would rather read that number than guess it.