Red teaming

Normal testing checks that a system does what it should. Red teaming checks what it can be pushed into doing. Red teamers try jailbreaks, prompt injections hidden in documents and web pages, attempts to extract secrets or other users' data, abuse of tools, and requests that should be refused.

For an agent, focus on what is unique to your setup: the tools it holds, the data it can reach, and the untrusted content it reads. The most valuable findings are usually paths where outside content leads to a real action, such as sending an email or changing a record.

Turn each successful attack into a test case. Adding red-team findings to your eval set means a later change to the prompt, model, or tools cannot quietly reopen a hole you already fixed.