Most AI trials end with an opinion. The team liked the tool, or did not. The sponsor saw a good demo. Weeks later finance asks whether it was worth continuing, and nobody can answer, because nobody wrote down how the work looked before.
This article is for a project lead or a finance partner who has a small AI trial running or planned: a drafting assistant, a document classifier, a support triage helper. It describes how to measure that one trial with three numbers. It is not about forecasting returns in general; for that, see what to expect from AI ROI.
Start before the trial, not after
The most common mistake is to start measuring when the tool goes live. By then the old way of working is gone, and the only comparison left is memory.
Two weeks before the trial, pick one process and one kind of work item, for example "incoming supplier invoices" or "first replies to support tickets". Write down in one sentence what a finished item looks like. Then collect the three numbers below for the current way of working. If the process has weekly peaks, make it three weeks.
Number 1: handling time per item
Measure how long one item takes from the moment someone starts on it to the moment it is finished. Do not use the time the item waits in a queue; the tool will not change that, and it hides the effect.
A simple method works. Ask three or four people to note start and finish times for twenty items each, using a shared sheet. Use the median, not the average, because a few very long items distort the average.
Be honest about what is included. If the tool saves five minutes of typing but adds three minutes of checking its output, the saving is two minutes. Count the checking.
Number 2: errors found
Define what counts as an error before you look at any results: a wrong amount, a missing field, a reply that had to be rewritten, an item sent back by the next team. Pick a short list of three to five error types and keep it fixed.
Take a random sample of items, around fifty before and fifty during the trial, and have someone who did not do the work check them against the list. Report the share of items with at least one error. Where the tool produces a draft that a person corrects, check the final result, not the draft; the final result is what the customer receives.
Number 3: total cost of the work
Time and errors only matter to finance when they become cost. For the same period, add up:
- the staff time spent on the items, at an internal hourly figure your finance team already uses;
- the cost of rework, meaning the time spent fixing errors found later;
- the cost of the tool itself, including licences, usage charges, and the hours spent setting it up and supervising it.
Divide by the number of items finished. The result is a cost per item, before and after. Do not include the tool's setup time in only one of the two periods; if the trial needed ten hours of preparation, that belongs in the trial cost.
Keep the comparison fair
Three things quietly spoil many comparisons.
Different work. If the trial period had simpler items, the numbers improve without any help from the tool. Compare items of the same type, or split them into simple and complex and report both.
Different people. The enthusiastic volunteers who join a trial are often the fastest on the team. Where you can, let the same people work in both periods.
A learning effect. People get faster at any new routine. Run the trial long enough, usually at least four weeks, to see whether the first-week gain holds.
Read the result
You now have three pairs of numbers. Four outcomes are common:
- Time down, errors flat or down, cost down: the trial paid off. Decide whether to extend it to more work.
- Time down, errors up: the saving moved the work to the next team. Look at where the errors appear before you continue.
- Time flat, errors down: the tool improved quality rather than speed. That can still be worth paying for, but say so.
- Nothing clearly changed: stop, or change one thing and measure again. An honest "no effect" is a useful result.
If the numbers are mixed, the next step is a decision about whether to expand, adjust or stop; see reviewing an AI agent pilot for a way to structure it.
A worked illustration
Imagine a team that answers first-line support emails. In the two weeks before the trial, they record the median handling time, sample fifty replies for errors and calculate the cost per reply from staff time. During a four-week trial of a drafting assistant, they repeat the same three measurements on the same type of email, with the same four agents.
Suppose the median time falls, the share of replies needing a later correction stays about the same, and the cost per reply is lower even after adding the tool's cost and the hours the team lead spent reviewing drafts. That is a result a finance partner can accept. Suppose instead the cost is lower but corrections rise; then the report says the saving is uncertain. This example is invented for illustration and does not describe a real client.
Write it on one page
The report should fit on a single page:
- What was tested, on which work, for how long, with whom.
- The three numbers before and after, with the sample sizes.
- What you could not measure or what may have distorted the result.
- A recommendation: continue, adjust or stop, and the next review date.
The third point matters most for trust. A result with a clear list of limits is easier to believe than a perfect one. If you want a model for the longer financial view once the trial has proved its value, see a one-page AI project payback model.
A short checklist
- One process and one kind of work item are chosen and described.
- The current process was measured for at least two weeks before the trial.
- Error types are defined before results are looked at.
- Setup and supervision time are counted as trial cost.
- The same type of work and, where possible, the same people are compared.
- The result, its limits and the decision fit on one page.
Sources
- Our related articles linked above.
- The approach described is a general working practice, not a standard; adapt it to your organisation. The example is invented and does not describe a real client. No figures or client results are claimed in this article.
