Vision Agents: Images, Screenshots, and PDFs
Give agents eyes: sending images and PDFs to models, reading charts and screenshots, and the mistakes vision models still make.
Everything your agents have done so far has gone through text: files, web pages stripped of their HTML, tool results as strings. But much of the world's information isn't text. It's in screenshots, scanned forms, charts in slide decks, photos of whiteboards, and PDFs whose meaning depends on layout. And much of the world's software is operated by looking at a screen and clicking.
This module, Beyond Text, extends your agents past that boundary. It builds on the agent from Module 5 and the multi-agent patterns from Module 6 (subagents in particular, which keep image-heavy work out of a main agent's context). You'll start here with vision: sending images and PDFs to a model, giving your agent a tool to look at files itself, and the mistakes vision models still make. Then you'll build document pipelines, browser and computer-use agents, and voice agents, and finish with the safety controls that agents acting on screens require.
What this module covers, and what changes
The six lessons in this module build on each other:
- Vision (this lesson): images, screenshots, and PDFs as input.
- Document processing pipelines: turning piles of documents into validated, structured data.
- Browser agents: agents that navigate and operate websites.
- Computer-use agents: agents that operate full desktop apps from screenshots.
- Voice agents: real-time speech in and out.
- Safety for agents that see and click: the extra risks, and the controls that contain them.
Three things change once agents leave text behind. Inputs get expensive, because an image costs far more tokens than a sentence describing it. Errors get quieter, because a misread digit looks exactly as confident as a correct one. And attacks get easier to hide, because instructions can live inside a picture where no human reviewer would think to look. Keep those three in mind through every lesson.