A small team tries an AI tool for a month. At the end, two people say it saves time and one says it made mistakes. The meeting turns into an argument about impressions, and the decision falls to whoever speaks last. A short scoring exercise prevents this. It takes about half a day, needs no software and gives the meeting something to look at.
This article is about scoring a pilot of a tool that helps with everyday work: drafting replies, summarising documents, sorting requests. If you are still choosing what to pilot, start with how to choose your first AI use case. If your tool is an agent that you will keep changing, the fixed set described in the AI agent test set of twenty real cases covers regular re-testing. Here the question is narrower: after the pilot, is the result good enough to go on?
Choose twenty cases before the pilot starts
Take twenty real items from the last one or two months of the work the tool will do. Real means real: the actual customer emails, the actual invoices, the actual meeting notes, with personal data removed or handled under your normal rules.
Mix them on purpose:
- twelve ordinary cases, the kind that make up most of the work
- four awkward cases, such as a vague request, a long thread or a missing attachment
- four cases where a mistake would be costly, for example a refund, a deadline or a legal wording
Choose them before anyone sees the tool's output. If you pick cases afterwards, you will drift towards the ones that went well. Put the list in a shared sheet and do not change it during the pilot.
Write the correct answer first
For each case, the person who normally does the work writes what a good result looks like. It does not have to be a full text. For a reply it can be three lines: the facts that must appear, the commitment that must not be made and the tone. For a summary it can be the five points that must be present.
This step is the most useful part of the exercise. It often shows that two colleagues disagree about what "correct" means, which is a finding on its own. Resolve it now, because a tool cannot be scored against a standard nobody has agreed on.
Score with three levels
Run the tool on the twenty cases and give each result a score:
- 2 usable as it is, or with a change a person makes in under a minute
- 1 useful starting point, but needs real editing or a second look
- 0 wrong, unsafe or not worth reading
Three levels are enough. Finer scales invite debate about the difference between a 6 and a 7. Ask a second person to score at least the awkward and costly cases independently. Where two scorers differ by a full level, discuss the case and note why.
Set the pass mark and the must-not-fail rule in advance
Write both numbers before you look at any results. A reasonable shape for a first pilot is:
- Pass mark: at least 14 of the 20 cases score 2, and no more than 2 score 0.
- Must-not-fail rule: none of the four costly cases may contain a wrong commitment, a wrong amount or an invented fact. One such failure means the tool is not ready for that kind of work, whatever the total.
These figures are an example, not a standard. Your numbers depend on how much a mistake costs and how closely the output is checked afterwards. A tool whose output is always reviewed by a person can pass with a lower bar than one that sends text straight to customers. What matters is that the line exists before the results do.
Also record the baseline: how long the same twenty cases take a person without the tool. Without it you cannot say whether a score of 2 actually saves anything.
Sort the misses by cause
Every case that scores 0 or 1 gets one label:
- Bad input. The source was incomplete or contradictory. A person would also have struggled.
- Unclear instruction. The tool was never told something the team takes for granted, such as which tone to use or which topics need a person.
- Tool limit. The tool cannot do this kind of task reliably, even with a clear instruction.
- Missing rule. The work itself has no agreed answer, so no tool could be right.
The labels decide what to do next. Bad input and missing rules are fixed by the team, not the tool. Unclear instructions are fixed by rewriting the instruction. Only the third kind is a reason to doubt the tool. Teams that skip this step tend to blame the tool for problems that were already in the process.
Make the decision
Compare the total with the pass mark, check the must-not-fail rule and look at the labels. Then pick one of three outcomes:
- Go on. The pass mark is met and the costly cases are clean. Decide who reviews output and how often you will re-score a fresh sample.
- Change and rerun. The pass mark is missed, but most misses are instructions, input or missing rules. Fix those, then run the same twenty cases again. Use the same cases so the comparison is fair, and rerun only once or twice. A pilot that is retried indefinitely is a hope, not a test.
- Stop. The misses are mostly tool limits, or the must-not-fail rule was broken. Write down what you learned and what would have to change for a second attempt.
Put the decision on one page: the date, the twenty-case total, the number of 0s, the label counts and the sentence "we decided X because Y". Someone who joins the team six months later should understand it without asking. For what comes after a longer pilot, see reviewing an AI agent pilot: expand, adjust or retire.
Common mistakes
- Choosing cases after seeing the output.
- Letting the tool, or the person who built the instruction, write the correct answers.
- Moving the pass mark after the results arrive.
- Averaging away a single dangerous failure.
- Treating every miss as the tool's fault.
Checklist
- Twenty real cases chosen before the pilot, with awkward and costly ones included
- A correct answer written for each case by the person who does the work
- A 2 / 1 / 0 score, with a second scorer on the hard cases
- Pass mark and must-not-fail rule written before the results
- Baseline time per case recorded
- Every miss labelled by cause
- A one-page decision: go on, change and rerun, or stop
Sources
- Common practice for evaluating a small tool trial, offered as a practical guide. The numbers (twenty cases, the 2 / 1 / 0 scale, the example pass mark) are starting points to adapt to your own risk. No client results or market figures are claimed in this article.
