Managing Prompt and Model Versions
Treat prompts and models like code: versioning, safe rollouts of new models, deprecation planning, and comparing versions with evals.
Someone tweaks one sentence in the system prompt on a Friday afternoon. By Monday, refund requests are being escalated twice as often, and nobody can say what changed or when. A month later, a new model comes out that's better and cheaper. You switch the model string, the evals look fine, and in production the agent starts calling a tool it used to ignore. Then the email arrives: the model you still use for classification will be retired in two months.
Prompts and models are the most influential code in your agent, and the least controlled. This lesson shows you how to treat them like code: version the prompt, tools, and model together, compare versions with evals before shipping, roll out new versions gradually with a fast rollback, and plan for model deprecations long before they become emergencies.
What counts as a version
The monitoring lesson introduced an agentVersion field on every run summary. Here's what it should capture. An agent's behavior is determined by more than its prompt:
- System prompt and any templates it's built from
- Tool definitions: names, descriptions, and schemas (descriptions are prompts too)
- Model ID
- Request parameters: effort,
max_tokens, thinking settings - Code that shapes context: what gets retrieved, how history is compacted, which tools are offered when
Change any one of these and you have a different agent. So version them together, as one immutable bundle with one ID. "Prompt v7 on model X with tools v3" invites confusion. "support-agent-v14" doesn't.