Measuring the ROI of Agents

Sooner or later, someone with a budget asks the question: are these agents actually worth it? The honest answer in most organizations is "we think so, but we can't really tell." Usage is up, people say they like it, and there's a slide claiming thousands of hours saved, built on a survey and a multiplication.

That answer won't survive a tough budget review, and more importantly, it can't tell you which agent work to expand and which to stop. This lesson shows you how to measure the return on agents properly: collecting a baseline before you start, choosing time and quality metrics that hold up, counting the full cost of ownership, comparing fairly, and reporting results in a way that builds trust instead of eroding it.

Why most AI ROI claims don't hold up#

ROI (return on investment) is the value you get compared to what you spend. For agents, both sides of that ratio are easy to get wrong. The most common errors:

• No baseline. Without knowing how long a task took before, "it now takes 10 minutes" means nothing.
• Self-reported savings. People remember the impressive runs and forget the re-prompting, checking, and fixing. Surveys of "hours saved" tend to run high.
• Ignoring review time. Producing a draft in two minutes doesn't help if checking it takes thirty.
• Counting time that isn't reallocated. Ten minutes saved per person per day may simply dissolve into the day. Saved time only has value if it turns into something: more output, faster turnaround, less overtime, or better quality.
• Cherry-picking. Reporting the best workflow and staying quiet about the four that didn't work.
• The novelty effect. Early enthusiasm inflates usage and effort in the first weeks. Measure after that settles.
• Ignoring quality. Faster but worse isn't a win. Neither is faster with more errors that someone downstream has to fix.

You won't eliminate all of these, but you can design around most of them.