Most companies decide once whether an AI agent is worth buying and never ask again. Yet an agent changes over time. The work around it changes, prices change, the documents it relies on get old, and people slowly stop checking its output. A short monthly review is the cheapest way to notice this while it is still easy to fix.
This article is for the owner or manager who is responsible for an agent that already works in the company. It does not repeat how to set a baseline before launch, which is covered in measuring the value of an AI agent, or the wider routine in maintaining an AI agent after launch. Here the focus is a single page you can fill in within an hour and a clear decision at the end.
Why monthly, and why only one page
A month is long enough to show a trend and short enough that a problem has not yet grown into a habit. A quarter is too slow for a young agent, and a week produces too much noise for a small company.
One page matters even more than the interval. A long report is written once and then ignored. A page with the same six numbers every month can be compared with the last one in two minutes. The NIST AI Risk Management Framework describes this as ongoing measurement and management of an AI system after it is deployed, not a one-time check (NIST: AI Risk Management Framework). A small company can follow the same idea with far less paperwork.
The six numbers
1. Cases handled
How many cases did the agent take this month, and how many did it finish without a person? Take this from the activity log (see what to record about an agent's actions), not from the vendor's dashboard alone. If the agent's own report and your records disagree, find out why before anything else.
2. Time saved, honestly counted
Multiply the finished cases by the average time a person needed for such a case before the agent. Use the baseline you measured at the start. Then subtract the time your people spend because of the agent:
- checking its output,
- correcting mistakes,
- taking over handed-over cases,
- answering customers who were confused by it.
What remains is the real saving. It is often half of the number the agent seems to produce, and that is normal. A saving that holds up after this subtraction is worth more than a large one that does not.
3. Errors
Count the mistakes that reached a customer or changed a record, and the ones your people caught in time. Split them into a few types, such as wrong information, wrong action and missing action. One bad month is a reason to look closer. Three months in the same direction is a trend and needs a decision.
4. Handovers
Record how many cases the agent passed to a person and for which reasons (see when an agent should hand the case to a person). A rising number may mean the documents are out of date or the work has changed. A number falling to almost zero may mean the agent has started to guess. Both deserve a look.
5. Customer and staff reaction
You do not need a survey. A few things are enough: complaints that mention the agent, customers who asked for a person, and one short question to the colleagues who work with it. "What annoyed you about it this month?" gives more than a score from one to five. If you told customers about the agent (see what to tell customers), check that the notice is still true.
6. Total cost
This is where small companies most often undercount. Add up everything for the month:
- licence or subscription fees,
- usage charges, which often grow with the number of cases and can change when the supplier changes its pricing,
- the share of any setup or integration cost, spread over a sensible period,
- hosting, tools and connections the agent depends on,
- the hours of the person who owns the agent and looks after its documents and settings, priced at their real hourly cost.
The last item is the one people forget. An agent without an owner's time soon starts to produce the problems described above.
Putting benefit and cost side by side
Once you have the numbers, the calculation is simple:
- Benefit: real time saved, multiplied by the hourly cost of the people whose time it frees, plus any other value you can show, such as faster replies that kept a customer.
- Cost: the total from the sixth number.
- Result: benefit minus cost, and the same figure for the last two months.
Do not hide a loss in a good story. If the result is negative for a month, write it down. A young agent may reasonably cost more than it saves for a while. What matters is whether the direction is right and whether you know when it should turn positive. The article on measuring the return on AI investments goes further into the longer-term picture.
Value that is not in money should be named, but kept separate. Fewer mistakes in a boring task or a more even quality of replies are real benefits. Just do not add them to the total with an invented price.
The one-page template
You can copy this directly:
- Month and owner of the agent.
- Cases: handled, finished without a person, handed over.
- Time: saved by the agent, minus time spent on checking and corrections, equals real saving.
- Errors: number, types, and what was done about each.
- Reaction: complaints, requests for a person, comments from colleagues.
- Cost: each item and the total.
- Result compared with the last two months.
- Changes in the agent, the documents or the supplier's terms since the last review.
- Decision: keep, adjust or stop, who does what, and by when.
Three possible decisions
Keep
The result is positive or moving the right way, errors are stable or falling, and nothing alarming is in the handovers. Write down that you checked, and leave it alone. A review that ends with "no change" is a successful review.
Adjust
The agent has value, but something specific drags it down: a task it does badly, a document that is out of date, a trigger for handover that is too loose, or a cost item that grew. Choose one or two changes, name the person responsible and test them as you would any change to the agent (see changing an AI agent after launch). Check the effect in next month's review.
Stop or reduce
The result has been negative for several months with no clear reason to expect improvement, or the errors are costing more than the agent saves, or the work it was meant for has disappeared. Stopping is not a failure. It saves money and attention for a better use. Do it in an orderly way, with its access removed and its data handled properly, as described in retiring or replacing an AI agent safely.
If you cannot decide, set a short deadline, for example two more months, and state in advance what number would make you stop.
Common mistakes
- Trusting the vendor's savings figure. It measures what the agent did, not what your company gained.
- Leaving out the owner's time. Someone is always looking after the agent, and that time costs money.
- Looking only at the average. A few serious errors can matter more than many small successes. Read a handful of real cases each month.
- Skipping the review when things seem fine. That is when attention slips and quiet problems begin.
- No decision at the end. Numbers without a decision are just a report.
A short checklist
- The owner of the agent is named, and the review is in the calendar.
- The same one page is filled in every month, using your own records.
- Time saved is reduced by checking, correction and handover time.
- Total cost includes usage charges and the owner's hours.
- Errors and handovers are read as trends and reasons.
- Each review ends with keep, adjust or stop, a person and a date.
If you have never reviewed your agent, do the first one now, using whatever numbers you can find. A rough page this month is more useful than a perfect one planned for later, and next month's page will already be better.
