Controlling Cost and Latency

Agents are expensive in a way single prompts aren't, and the reason is structural. Every turn of the agent loop re-sends the entire conversation: the system prompt, the tool definitions, every previous message, every tool result. A 30-step run sends your system prompt 30 times. Costs grow faster than the work, and so does latency, because the model re-reads everything before each step.

The good news is that the biggest levers are well understood and mostly mechanical. This lesson covers the four that matter most: prompt caching, model routing, batch processing, and budgets, plus the latency techniques that make agents feel fast. Used together on the right workload, they can cut costs by an order of magnitude. You'll measure a baseline first, because the only savings that count are the ones you can see.

Where the money and time actually go#

Before optimizing, look at real numbers. Every API response includes a usage object that tells you exactly what you paid for:

import Anthropic from "@anthropic-ai/sdk";
 
export function logUsage(runId: string, step: number, response: Anthropic.Message) {
  const u = response.usage;
  console.log(
    JSON.stringify({
      runId,
      step,
      model: response.model,
      input: u.input_tokens,                         // uncached input
      cacheWrite: u.cache_creation_input_tokens ?? 0, // written to cache this call
      cacheRead: u.cache_read_input_tokens ?? 0,      // served from cache
      output: u.output_tokens,
      stop: response.stop_reason,
    }),
  );
}

Log this for every model call in a few dozen real runs, and patterns jump out. For most agents:

• Input tokens dominate cost. Output per step is small, but the input (the whole history) is large and resent every step.
• Output tokens dominate latency. Models read input quickly but generate output one token at a time. A step that writes 2,000 tokens is slower than one that reads 50,000.
• A few runs cost far more than the rest. Long runs, retries, and loops form a long tail. Your average cost hides them; look at the 95th percentile too.
• Tool time is often invisible. Slow APIs, web fetches, and test suites can account for more wall-clock time than the model.

Now you know which lever to pull first.