Most AI agents are tested the same way: someone tries a few examples, the answers look good, and the agent goes live. Then the first unusual request arrives and nobody knows whether the result is a surprise or normal behaviour.
This article is for an operations or technology lead who has to decide whether an agent is ready to start work, and later whether a change made it better or worse. The tool is simple: a fixed set of about twenty real cases that you run every time.
Why twenty, and why a fixed set
Twenty is not a magic number. It is enough to cover the common situations and several awkward ones, and small enough that a person can read every answer in an hour. A set of five hides problems; a set of two hundred is never run properly.
The set has to be fixed. If you pick new examples each time, you cannot tell whether the agent got better or you simply chose easier cases. With the same cases, a change in the result means a change in the agent.
Where the cases come from
Use your own history. Open the mailbox, the ticket system or the order log of the process the agent will handle and pick real items, with personal data removed or replaced.
A workable mix for twenty cases:
- Ten typical cases. The ordinary requests that make up most of the volume. If the agent fails these, it is not ready.
- Four awkward inputs. Missing information, a mistake in the request, two requests in one message, an unusual way of phrasing.
- Three cases outside the agent's job. Things it should hand to a person or decline, such as a complaint, a legal question or a request for an exception.
- Three risky cases. Requests where a wrong action costs money or trust, such as a refund above the agent's limit or a message to the wrong customer.
Ask the people who do the work today which cases annoyed them most last quarter. Those are usually the best additions.
What counts as a pass
Write down what a good result looks like before you run the case. One sentence is enough: "Replies with the delivery date from the order system and does not promise a refund", or "Passes to the account manager and does not answer the question itself".
Three rules keep the judging honest:
- Judge the outcome, not the style. A correct answer in clumsy wording passes; a smooth answer with a wrong date fails.
- Use three marks only: pass, fail, unclear. Unclear cases are discussed, then the sentence is improved.
- Someone other than the person who built the agent does the judging, or at least a second person reads the failures.
A small example
Imagine an agent that answers customers' questions about delivery. A fragment of the set could look like this (the example is invented):
| Case | Input | Expected |
|---|---|---|
| 3 | "Where is my order 4471?" | Gives the status and date from the order system |
| 9 | "Order 4471 and also change my address" | Answers the status, passes the address change to a person |
| 14 | "I want my money back, this is the third delay" | Does not decide; passes to a person with the history |
| 18 | A message with no order number | Asks for the number, does not guess |
Notice that two of the four expected results are "pass it on". Knowing when not to act is part of the job, and the set should check it.
When to run the set
- Before launch. Agree the bar in advance, for example "all risky cases pass and at least eighteen of twenty overall". The numbers are your decision, not a standard.
- After every change to the instructions, the model, the connected data or the tools. Small changes break things in surprising places.
- After a real failure. Add the failing case to the set, fix the agent, and run everything again.
- On a calendar. Once a quarter even if nothing changed, because the outside world and the underlying model can change under you.
Keep a simple table of dates and results. A drop from nineteen to sixteen after a change is a clear signal to roll back or to look at which cases failed.
Who does what
- The process owner supplies and approves the cases and the pass sentences.
- The builder runs the set and fixes failures, but does not decide that a failure is acceptable.
- A second person judges the answers and signs off before launch.
Keep the set and the results in one shared place, with a name and a date on every run.
Common mistakes
- Using invented, tidy examples. Real requests are messier, and the mess is where agents fail.
- Tuning the agent to the set. If you keep adjusting until all twenty pass, you may have taught it the test. Add new real cases from time to time and watch whether the result holds.
- Judging by impression. "Looks fine" is not a result. Compare the answer with the written sentence.
- Letting the set grow without limit. Retire cases that no longer matter and keep it near twenty.
- Running it once. The value is in the repetition.
Checklist
- Twenty real cases from your own history, personal data removed
- A mix of typical, awkward, out-of-scope and risky cases
- One pass sentence per case, written before the run
- A second person judges the failures
- The bar for launch agreed in advance
- A rerun after every change and after every real failure
- Dates and results kept in one place
The set pairs naturally with a clear agent brief with an acceptance test, which defines what the agent should do, and with the habits in changing an AI agent after launch. If you are still choosing where to start, see choosing the first process for an AI agent. Once the agent is running, a monthly review is a good moment to ask which new real cases belong in the set.
Sources
- Our related articles linked above.
- The mix of cases, the number twenty and the example pass bar are a working practice, not a standard; adapt them to your process. The example is invented for illustration. No client results or market figures are claimed in this article.