Launch day is behind you. The agent answers tickets, drafts documents or sorts requests, and the team has stopped talking about it. Then a director asks a fair question: "What are we getting for this?" If the answer is a screenshot of a busy dashboard or a sentence like "people like it", the project is exposed. The next budget round will treat it as a cost.
This article is for the manager who owns the agent after launch. It assumes you already collected a baseline before the pilot, as described in how to measure whether AI agents really save work. Here the question is different: how do you report value regularly, to people who do not work with the agent every day, in a way that stays honest.
1. Use one page, and keep it the same every time
Leadership does not need every metric. It needs a small set of numbers it can compare month after month. One page with four parts is enough:
- Adoption: is the agent actually used for the work it was built for?
- Quality: is the output good enough, and is that getting better or worse?
- Cost: what does it cost to run, review and fix?
- Time saved: how much human effort did it remove, compared with the baseline?
Keep the layout fixed. When the format changes every month, readers cannot see trends and start to suspect that the numbers were picked to look good.
2. Pick a few numbers for each part
Choose two or three measures per part, not ten. Examples that work in many teams:
- Adoption: share of eligible cases that go through the agent; number of active users in the team; share of cases where staff bypass the agent and do it by hand.
- Quality: share of agent outputs accepted without changes; share corrected by a person; share escalated to a human; complaints or errors that reached a customer.
- Cost: model and platform cost per month; person-hours spent on review and correction; cost per handled case compared with the baseline.
- Time saved: median handling time compared with the baseline; backlog size; hours returned to the team and what they were used for.
Each number needs a clear definition written once: what counts as a case, what counts as a correction, who records it. A definition that lives only in one person's head will change when that person leaves.
3. Separate early signals from late results
Not every number is ready at the same time. In the first weeks you can see leading signals: usage, how often people accept or correct output, how many cases are escalated. Results such as cost per case, customer satisfaction or a shorter backlog appear later, because they need enough volume and a stable process.
Say this on the page. A line such as "cost per case: too early to judge, review at day 60" is more credible than a figure calculated from forty cases. The idea of watching a small number of indicators and setting explicit targets for them is familiar from reliability engineering, where Google's SRE practice recommends choosing a few service level indicators and agreeing objectives for them instead of tracking everything (Google SRE book). The same discipline applies to an agent: few indicators, agreed targets.
4. Set targets and thresholds before the first report
For every number, write down three values before you publish anything: where you started, where you want to be and the level at which you would act. For example: "Accepted without changes: baseline not applicable, target 80 percent, review the process if it falls below 65 percent for two weeks."
This protects you in both directions. You cannot quietly move the goal when the first month disappoints, and nobody can say you underdelivered against a target you never set. It also turns the report into a decision tool, because a number below the threshold has a pre-agreed consequence.
5. Show the whole cost, not only the licence
The most common mistake in an agent report is counting the saving and leaving out the effort that produces it. Put these next to the saving:
- Running cost: model usage, platform and integration fees.
- Review time: the minutes a person spends checking output that is approved.
- Rework: the time spent fixing output that was wrong.
- Maintenance: updates to prompts, knowledge sources and rules, plus time spent on incidents.
If you set cost limits and loop limits for the agent, as described in cost controls, budgets and loop limits, the running cost is already measurable. The human cost is usually the one that is missing. A rough estimate is acceptable as long as it is labelled as an estimate.
6. Say what is measured and what is estimated
Time saved is often calculated, not observed: cases handled multiplied by the minutes a person used to need. That is a reasonable method, but it is an estimate, and readers should know it. Mark each figure as measured, sampled or estimated, and give the method in one line. If you sampled fifty cases to compare handling time, say that.
The NIST AI Risk Management Framework treats measurement as its own function and notes that metrics should be documented together with their limits and uncertainty (NIST AI RMF Playbook, Measure). You do not need the full framework to apply the habit: a short note under the number saying how it was obtained is enough.
7. Watch for the traps
- Vanity numbers. Messages processed or tokens used show activity, not value. Replace them with cases resolved and accepted.
- Counting savings twice. If an hour is saved by the agent and also credited to a process change, it appears in two reports. Agree who owns each saving.
- Time saved that goes nowhere. Ten hours a week saved is only a result if those hours were used for something: more complex cases, shorter queues, fewer overtime hours. If nothing changed, say so and decide what the time should be used for.
- Averages that hide problems. A good average handling time can sit on top of a small group of cases the agent handles badly. Show the share of escalations and a few of the worst examples.
- Survivorship. If staff avoid the agent for hard cases, the cases it does handle look easy and its quality looks high. Report the bypass rate.
- Measuring people instead of the process. If staff think the scorecard rates them, they will stop reporting errors. Keep the agent's metrics separate from individual performance, as discussed in explaining an AI agent to your team.
8. Use a 30, 60 and 90 day rhythm
A fixed rhythm keeps expectations realistic:
- Day 30: adoption and quality. Is the agent used, and is the output usable? Fix obvious problems; do not judge value yet.
- Day 60: add cost and time saved with enough volume to be meaningful. Compare with the baseline and with the targets.
- Day 90: a decision meeting. Based on the scorecard, choose to expand, adjust or stop.
The last step is the one teams skip. A report that ends with a chart and no decision does not change anything. The decision options are covered in reviewing an AI agent pilot: expand, adjust or retire; after the first 90 days the same three choices apply, and your scorecard should be the evidence behind whichever one you pick.
A short checklist
- Is there a one-page scorecard with adoption, quality, cost and time saved, in the same format each month?
- Does every number have a written definition and a named person who records it?
- Do you have a baseline, a target and a threshold for each number, set before the first report?
- Are early signals separated from late results, with "too early" stated where it applies?
- Does the cost side include review, rework and maintenance?
- Is every figure marked as measured, sampled or estimated?
- Does each review end with a decision to expand, adjust or stop?
An agent that quietly does useful work is easy to forget and easy to cut. A short, honest, regular report is what lets leadership see the value that is really there, and see early when it is not.
