Prompt Caching and Cost Tracking in Your Agent

Here's a fact about your agent that's easy to miss: every turn resends everything. The system prompt, all five tool definitions, the memory tools, the task, and every message so far go back to the API on every single model call. On a ten-turn run you pay for your system prompt ten times, and the history from turn one gets billed again on turns two through ten.

In this lesson you'll fix both halves of that problem. You'll add prompt caching, so the repeated part of each request is billed at a steep discount and processed faster. Then you'll track exactly what each run uses, convert it to dollars, log it, and enforce a cost budget that ends runs gracefully. You'll finish with an agent that tells you, after every run, what it cost and how much of its input came from the cache.

Where an agent's tokens go#

Picture the requests in a five-turn run. Each one is the previous request plus a little more:

turn 1:  [tools][system][task]
turn 2:  [tools][system][task][reply 1][results 1]
turn 3:  [tools][system][task][reply 1][results 1][reply 2][results 2]
turn 4:  [tools][system][task][reply 1][results 1][reply 2][results 2][reply 3][results 3]
turn 5:  ...and so on

Almost everything in each request is a repeat of the last one. That's the shape that makes caching so effective for agents: the prefix, meaning everything from the start of the request up to some point, is identical from turn to turn, and only the tail is new.

Output tokens are the other cost. They're priced several times higher than input per token, and on current models they include thinking, the reasoning the model does before answering. You can't cache output, but you can measure it, and you'll do that below. Context Windows, Tokens, and Cost covers the basics if any of this is new.