Computer-Use Agents: Operating a Desktop

Some work lives in software that has no API, no web version, and no export button: an old accounting package, a desktop design tool, a line-of-business app that only runs on one Windows machine in the back office. For a long time, the only automation option was screen-scraping macros that broke when a dialog moved ten pixels.

Computer use lets a model operate a whole desktop the way a person does: look at a screenshot, move the mouse, click, type, and look again. It's the most general tool an agent can have, and also the slowest, most expensive, and riskiest. In this lesson you'll learn how computer use works under the hood, what it handles well and badly today, how to set up a disposable sandbox desktop, and how to write a working agent loop with Anthropic's computer use toolset.

How computer use works#

Computer use is the browser agent loop with the structure taken away. A browser agent can read the page's accessibility tree; a computer-use agent usually gets only pixels.

1. Your code takes a screenshot of the desktop and sends it to the model.
2. The model looks at the image, decides what to do, and returns an action with coordinates: "left click at (412, 288)", "type 'Q3 report'", "press ctrl+s".
3. Your code performs the action on the machine, takes a new screenshot, and sends it back.
4. Repeat until the task is done.

The model never touches the machine. It's a client-side tool, like every tool you've built: the model proposes actions and your code executes them. That's what makes it possible to run computer use safely. You decide which machine it acts on, and you can refuse any action.

Coordinates are in the pixel space of the screenshots you send, measured from the top-left corner. If you downscale screenshots to save tokens, you must scale coordinates back up before clicking. Getting this wrong is the most common reason a new computer-use agent "clicks in the wrong place."

Anthropic's computer use toolset#

In the Claude API, computer use is a predefined tool. For current models (the Claude 5 family) you declare it with a single entry, { type: "computer_toolset_20260801" }, with no name and no display size. The toolset contains member tools, including screenshot, left_click, double_click, type, key, scroll, wait, and zoom (which lets the model look at a region of the screen at higher detail). Each action Claude takes arrives as a tool_use block whose name is the member and which carries toolset_name: "computer". Claude often batches several actions in one turn; you run them in order and return one tool_result per action, each echoing the toolset_name.

Older models use an earlier, beta version of the tool with a different request shape (a single computer tool with display dimensions and an action field). The 5.5-generation models reject that older version on the Claude API, so if you find a tutorial using it, check the computer use tool page in Anthropic's docs for the version that matches your model and platform. OpenAI and Google offer computer-use capabilities in their APIs too; the request shapes differ, but the screenshot-act-screenshot loop is the same everywhere.