Red-Teaming Your Agent

You've threat-modeled your agent, layered injection defenses, sandboxed it, and put humans in front of risky actions. Do the defenses actually hold? You won't find out by reading your own system prompt and nodding. You find out by attacking the agent the way a motivated adversary would, on your own terms, before someone else does it on theirs.

That's red-teaming: a structured, authorized attempt to make your own system fail. This lesson shows you how to turn your threat model into an adversarial test set, run attacks through the eval harness you built in Module 9 (the triage-evals project and its agent-eval.ts runner), run a focused red-team exercise with people, and fix what you find at the right layer. You'll finish with an attack suite that runs in CI, so every defense you add stays added.

What red-teaming is, and what it isn't#

Your eval suite checks behaviors you already know about. Red-teaming hunts for the ones you don't. The two work together: red-teaming finds a failure, the failure becomes an eval case, and the eval case makes sure the fix stays fixed.

Red-teaming an agent is defensive work, and it needs ground rules. Before you start, write down:

• Scope. Which agent, which version, which environment. Attack a staging deployment or the fake-world harness, never production systems or real customer data.
• Payloads. Use harmless, observable markers: a canary string that must never appear in output, a refund that only exists in a fake world, a word like PINEAPPLE that proves an instruction was followed. You're testing whether the agent can be steered, not trying to cause damage.
• Authorization. Red-team only systems you own or have written permission to test. Third-party tools and APIs your agent calls are out of scope unless their owners agree.
• Handling. Findings are security vulnerabilities. Store them where only the team fixing them can see them until they're fixed.