Safety for Agents That See and Click
The extra risks when agents read screens and operate interfaces — injected instructions in pages, misclicks, and purchases — and the controls that contain them.
A text agent with a send_email tool is easy to reason about: you can see every call, check the recipient, and require approval. Now give an agent a browser or a desktop. It reads pages written by strangers, clicks at coordinates that don't say what they'll do, and operates sites where you may already be logged in with a saved credit card. The same agent that books your meeting room can, with one misread or one planted sentence, buy something, submit a form with your personal details, or accept terms on your behalf.
This lesson closes Module 7 by putting the extra risks of screen-operating agents in one place, and the controls that contain them. You'll leave with a layered defense you can apply to the browser and computer-use agents you built earlier in this module, a working navigation guard and confirmation checkpoint, and a checklist to run before any agent clicks on anything that matters.
Why screens raise the stakes
Everything you learned about tools still applies, but four things get harder when the interface is a screen.
Everything on screen is input from strangers. A browser agent reads web pages, ads, reviews, search results, and emails rendered in a webmail tab. A computer-use agent reads whatever pixels appear, including text inside images. Any of it can contain instructions aimed at the agent. This is prompt injection, which Module 10 covers in depth in prompt injection and how to defend against it. On screens it's worse because the attack surface is the whole visible web, and text can be hidden in places a human wouldn't notice: white-on-white, tiny fonts, image alt text, or a banner that appears only to automated browsers.
Actions are opaque. "Click at (840, 612)" doesn't tell your code whether that's "Cancel" or "Confirm purchase." With a structured tool you can inspect parameters before running them. With pixel-level actions you often can't, so policy has to work at other layers: where the agent can go, what it can type, and what accounts it holds.
Authority is ambient. If the browser has your cookies, the agent is you on every site you're signed into. Ambient authority means permissions that come from the environment rather than from anything the task explicitly granted, and it's the most dangerous property a screen agent can have.
Mistakes need no attacker. Misclicks, misread numbers, a dialog that moved, a cookie banner that covered the real button. A model that's right 99 times out of 100 on a single click is not reliable across a 40-step task.