Caching

Reading it all again

Chapter 1 showed the loop: the model keeps nothing between calls, so the harness sends it the whole conversation every time. On a toy game that was a handful of calls. This chapter follows a longer session. We gave Claude Code a real bug report from the issue tracker of Black, a tool that formats Python code, along with Black's source code. The report's title:

The session made model calls in seconds. It was recorded on with . The conversation grew from tokens on the first call to on the last (a token is a word or a piece of one). Added up over all the calls, the model read tokens.

Most of that reading came from the cache: a stored copy of text the model's provider saw recently. The usage the API reports for each call splits what the model read into two parts: tokens read from that stored copy, and tokens that were new. The chart shows both, call by call.

From the second call on, each call found almost everything the previous call had read in the cache. Only what had been added since was new: between and tokens a call. Over the session, of everything the model read came from the cache.

Even the first call read of its tokens from the cache. This recording can't say where they came from. Probably from other sessions we recorded with Claude Code earlier that day.

What the cache does to the price

A token read from the cache isn't free, but it's cheap. Anthropic's pricing page lists these prices for , per million tokens: for normal input, for input read from the cache, and for what the model writes. A token read from the cache costs of a normal one.

The recording holds only the session's total cost, . From the prices and each call's token counts we get , the same to within a cent. So the per-call costs below are our arithmetic on the published prices, checked against that total. They aren't recorded.

Without a cache, the same calls would have cost , times as much. The gap widens as the session goes on, because every call reads the whole conversation and the conversation only grows. With the cache, the tokens the model wrote cost more than all its reading: of the total.

First it has to be written

The cache only holds what was put in it. In the chart above, the new tokens on each call are the ones written into the cache, so the next call can read them.

Writes come in two kinds, and the pricing page lists them separately: a copy kept for minutes, at per million tokens, and one kept for an hour, at . Every write in this session was the hour-long kind: tokens, against of the shorter kind.

Chapter 1's recordings show a write being read back. In its run , the first call read tokens from the cache: all were new. Run started minutes after run ended. Its first call read of its tokens from the cache, most likely the copy run had written.

What breaks it

To see what makes the cache miss, we ran a few small experiments of our own on . In a toy folder holding four files from Python's standard library, Claude Code (with ) read the files and described each one. Then we asked short questions, each one a new run of Claude Code that resumed the same conversation, and changed one thing at a time. The model was told to answer without using any tools, so each question is a single call.

The extra instruction didn't cost a cache miss, and the recording can't tell us whether it reached the model at all. A different model is a different matter: found little of the conversation in the cache and had to write most of it again. Back on , the cache still held its earlier copy. These are four questions on one small conversation, on one day. They show what happened, not why.