Jailbreak

Jailbreaks use role-play, hypothetical framing, encoded text, long manipulative setups, and many other tricks to talk a model out of its guidelines. Model developers train against known techniques, and attackers keep inventing new ones.

A jailbreak is different from prompt injection, though the terms are often mixed up. In a jailbreak, the person typing is trying to subvert the model. In prompt injection, the attack arrives through third-party content the agent processes, such as a web page or email, and the user may be the victim.

For agent builders, the practical lesson is to never rely on the model's refusals as your only control. Enforce important limits in code and permissions, so that even a fully jailbroken model cannot take actions or reach data it should not.