Computer use

With computer use, the model receives screenshots of a desktop and responds with actions such as moving the mouse, clicking, scrolling, or typing. Your code carries out each action, takes a new screenshot, and sends it back. It is the agent loop applied to a graphical interface.

This lets an agent work with applications that have no API or command-line interface. The tradeoff is speed and reliability: interpreting screens is slower and more error-prone than calling a tool that returns structured data.

Because a computer-use agent can do anything you could do at the keyboard, run it in an isolated virtual machine or container, keep sensitive accounts out of reach, and require confirmation before consequential actions.