Reliability Engineering: Rate Limits, Fallbacks, and Degradation

Your agent survives a crash and resumes where it left off. That doesn't make it reliable. On launch day, traffic triples and every run starts failing with 429 rate-limit errors. A week later the model API is overloaded for twenty minutes and your queue backs up for hours. Then the search API your agent depends on goes down, and every run spends its whole step budget retrying it.

Reliability engineering is the work of staying useful when the things you depend on misbehave. This lesson goes beyond the single-agent error handling from Module 5 to your whole fleet. You'll learn to read and respect rate limits, stop retries from multiplying into outages, add model fallbacks behind circuit breakers, set timeouts at every level, keep interactive traffic ahead of background work, and degrade gracefully instead of falling over. Then you'll inject failures on purpose to prove it all works.

Know what you're up against#

Start by naming the failures. The model API gives you a different signal for each, and each needs a different response:

That second row is easy to miss. On the Anthropic API, a spend cap returns the same rate_limit_error type as an ordinary rate limit, but it has no retry-after header and the error body carries error.details.error_code set to enforced_spend_limit_reached. Retrying it just burns time until the next billing month.

How rate limits work#

Anthropic measures limits per model class in three dimensions: requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM). Hitting any one returns a 429. A few details matter a lot for agents:

• Limits use a token bucket: capacity refills continuously rather than resetting each minute. A burst can exceed the limit even if your per-minute average is fine.
• For most current models, cached input tokens don't count toward ITPM. Only uncached input and cache writes do. The prompt caching from controlling cost and latency raises your effective throughput as well as cutting cost.
• max_tokens doesn't count against OTPM. Only tokens actually generated do, so a generous max_tokens costs nothing in rate-limit terms.
• A sudden sharp increase in usage can trigger acceleration limits. Ramp traffic up gradually, for example during a launch or a backfill.
• Responses carry headers such as anthropic-ratelimit-input-tokens-remaining and anthropic-ratelimit-requests-reset, so you can see your headroom before you hit the wall.

You can read those headers on every call with the SDK's withResponse():

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic();
 
export async function createWithHeadroom(params: Anthropic.MessageCreateParamsNonStreaming) {
  const { data, response } = await client.messages.create(params).withResponse();
  const remaining = Number(response.headers.get("anthropic-ratelimit-input-tokens-remaining"));
  const limit = Number(response.headers.get("anthropic-ratelimit-input-tokens-limit"));
  if (limit > 0 && remaining / limit < 0.1) {
    console.warn(JSON.stringify({ event: "rate_limit_low", model: params.model, remaining, limit }));
  }
  return data;
}

Graph that warning. When it fires regularly at peak, request a higher limit before your users find the ceiling for you.