Browser Agents: Automating the Web
Agents that click, type, and read websites: browser automation tools, when to use an API instead, and how to keep web agents reliable.
Much of the world's useful software only exists as a website. Your vendor's billing portal has no API. The government form has no export button. The internal admin panel was built in 2014 and nobody wants to touch it. For years, automating these meant writing brittle scripts that broke whenever a button moved.
Browser agents change that. An agent that can open pages, read them, click, and type can work through a website the way a person does, adapting when the layout shifts. They are also slow, expensive, and easy to mislead, so knowing when not to use one matters as much as knowing how. In this lesson you'll learn the three ways to give an agent a browser, how to decide between a browser and an API, and the habits that keep web agents reliable.
How an agent drives a browser
Every browser agent runs the same loop you learned in the agent loop: observe the page, decide on an action, act, observe again. The interesting differences are in how the agent sees the page.
| Way of seeing | What the model gets | Strengths | Weaknesses |
|---|---|---|---|
| Accessibility snapshot | A structured text outline of the page: headings, links, buttons, form fields, each with a reference ID | Cheap in tokens, precise ("click ref 14"), no vision needed | Misses things only visible in pixels: charts, canvas apps, visual layout |
| Screenshot | An image of the viewport, acted on with x/y coordinates | Works on anything a human can see | Costs far more tokens per step, and clicks can miss by a few pixels |
| Page text or HTML | The raw text or markup | Great for reading and extracting | Not enough on its own to operate the page |
The accessibility tree is the structured description of a page that browsers already build for screen readers. It lists every interactive element with its role ("button", "textbox") and name ("Submit order"). Because it's text, a model can read it quickly and refer to elements by ID instead of guessing coordinates. Most good browser tools lead with the accessibility tree and fall back to screenshots when a page needs eyes, which is why the lesson on vision still matters here.