Are your AI agents really saving work? How to measure it honestly

Three months after an AI agent goes live, someone asks the manager a simple question: is it working? The usual answer is a feeling. The team says it is faster, the vendor dashboard shows thousands of completed tasks, and nobody has checked what happens after the agent says "done".

This article is for the manager who has to answer that question with something better than a feeling. It does not cover how to cap spending or how to test reliability before launch. For those, see cost controls and loop limits and evaluating agent reliability. The focus here is the comparison: the work with the agent against the work without it.

Why feeling fast is not evidence

People are poor judges of their own speed with a new tool. In a 2025 randomised study by METR, experienced open-source developers worked on real tasks in their own repositories, with AI tools allowed on some tasks and not on others. They expected AI to make them faster and, after the study, still believed it had. The measured result went the other way: tasks with AI took longer on average. The study is small and covers one kind of work, so do not generalise it to your company. The lesson is narrower and more useful: perceived and measured speed can differ, and only measurement tells you which one you have.

The opposite result exists too. A study of customer support agents published by the US National Bureau of Economic Research found that an AI assistant raised the number of issues resolved per hour, with the largest gains for less experienced staff. Different work, different tool, different outcome. Both studies point to the same practice: measure your own process, in your own conditions.

Step 1: define the unit of work

Pick one repeatable unit, such as a support ticket, an invoice, a lead qualified, or a report section. Then define where it starts and where it ends.

• Start: when the request arrives, not when the agent picks it up.
• End: when the result is accepted and used, not when the agent produces it.

This single choice changes the picture. If you stop the clock when the agent finishes, every agent looks fast. If you stop it when a person has accepted the result, you capture the queue for review, the corrections, and the cases sent back.

Step 2: take a baseline before the agent starts

You cannot measure a saving without knowing the starting point. For two to four weeks before launch, or from recent history if you have it, record for the same unit of work:

1. Time per unit, from request to accepted result, as a median and as the slow end (for example the 90th percentile). Averages hide the slow cases that frustrate customers.
2. Hands-on effort per unit, in minutes of human work. Elapsed time and effort are different numbers.
3. Error or rework rate, meaning how often a result is corrected or sent back.
4. Volume, so you can see whether demand changed for reasons unrelated to the agent.

If no history exists, run a short manual period first. Rough numbers taken before launch are worth more than precise numbers invented afterwards.

Step 3: measure speed, quality, and effort together

One metric alone can be gamed, usually by accident. A small set, read together, is harder to fool.

The DORA research programme on software delivery uses the same idea: speed measures such as lead time are paired with stability measures such as change failure rate, because improving one while the other worsens is not progress. You can borrow the principle outside software. Always put a quality measure next to every speed measure.

Step 4: count the hidden costs

The saving on the agent's own work is the easy part. The costs people forget fall into four groups.

• Review time. If a person must check every output, the saving is the drafting time minus the checking time. For some tasks checking takes nearly as long as doing. Sample the real review minutes instead of guessing. Where approval is required by design, see human approval and audit trail.
• Rework and escalation. Every result that is corrected, sent back, or handed to a person after the agent has already spent time on it costs more than doing it directly. Track how many units fall into this group.
• Maintenance. Prompts, rules, connections and knowledge sources need an owner. Count the hours spent on them each month, plus the time spent on incidents.
• Running cost. Model usage, tools, licences, and hosting. These are usually the smallest and best-documented part, which is why they get all the attention.

A simple net figure keeps the discussion honest:

Net time saved per unit = baseline effort − (agent-assisted effort + review + rework share + maintenance share)

Multiply by volume, subtract running cost, and compare to the price of the time you recovered. If the result is close to zero, that is useful information too.

Step 5: look for the savings you are not measuring

Not every benefit is a saved minute. Some real gains are:

• Faster first response, even when total effort is the same.
• Coverage outside working hours.
• Consistency, such as every request being triaged by the same rules.
• Staff moving to work that needs judgement.

Treat these as separate, named benefits, and measure each one with its own indicator. Do not fold them into a time saving to make the case look stronger. For a wider view of return on investment, see how to measure the return on AI investments.

Step 6: watch for the common traps

• Measuring activity instead of outcomes. Tasks started, messages sent, or tokens processed say nothing about whether work was finished.
• Counting only the easy cases. If the agent takes the simple requests and people keep the hard ones, the average time per unit for people may rise even though total effort fell. Compare like with like, by type of request.
• Comparing with an idealised past. Use recorded baseline numbers, not how fast the team remembers being.
• Letting the people who built the agent measure it alone. Ask someone with no stake in the outcome to check the method. A shared dashboard that an operations lead owns beats a slide from the project team.

Step 7: agree the decision rule up front

Before launch, write down what result means what. For example:

1. Expand if net effort per unit falls by at least a set share and the rework rate is no worse than the baseline.
2. Adjust if speed improves but quality or review cost eats the gain. Narrow the scope, tighten the instructions, or add a check.
3. Stop or pause if after the agreed settling-in period the net effect is flat or negative.

Fix the review date too, for example at 30, 60, and 90 days. A written rule stops the common drift where a project continues because stopping it feels like admitting a mistake. If you are still deciding where to start, the first agent pilot playbook and the staged rollout guide describe how to run a pilot that produces this kind of evidence.

A checklist for the manager

• One defined unit of work, with a start and an end that includes acceptance.
• A recorded baseline for time, effort, rework, and volume.
• Speed and quality measured side by side.
• Review, rework, maintenance, and running cost all counted.
• Other benefits named and measured separately.
• A decision rule and review dates written down before launch.
• Someone outside the project team owns the numbers.

The honest answer to "is it working?" is sometimes "yes, and here is how much". Sometimes it is "partly, for these cases". A manager who can say either, with numbers, is in a far better position than one with a good feeling and a busy dashboard.

Sources

• METR: Measuring the impact of early-2025 AI on experienced open-source developer productivity
• arXiv: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
• NBER: Generative AI at Work (Brynjolfsson, Li, Raymond)
• DORA: The four key metrics
• NIST: AI Risk Management Framework