The supplier shows the new tool, everything works, and the demo ends with applause. Two weeks after go-live, the team keeps a spreadsheet on the side because "it does not handle our odd cases." The odd cases were never part of the demo, because the supplier chose the examples.
This article is for an operations lead who has to say yes or no to a new tool, whether it was built for you or bought. It describes a two-hour acceptance session run on real work, with a simple scoring rule and clear grounds for sending the tool back.
Why a demo is not an acceptance test
A demo is designed to succeed. The data is clean, the path is the shortest one and the person clicking knows exactly where everything is. Your daily work has missing fields, duplicate customers, urgent exceptions and people who are interrupted every ten minutes.
An acceptance session reverses the roles. Your people bring the cases and do the clicking; the supplier watches and answers questions. The goal is not to find every defect. It is to learn, before you sign off, whether the tool carries a normal working day.
Before the session: prepare the cases
Preparation takes about half a day and decides whether the two hours are useful.
- Pick eight to twelve real cases from the last month. Take them from the actual inbox, sheet or system the tool replaces. Include at least two awkward ones: a case with missing information and a case that needed a manager's decision.
- Remove or mask personal data. Use a test copy of the tool, or replace names and numbers with neutral ones, while keeping the shape of the case.
- Write each case on one line. For example: "Customer changes the delivery date after the order was confirmed." Add the result you expect, such as "order updated, warehouse notified."
- Fix the environment. The tool should be installed as it will be used: real accounts, real roles, the same browser or device your team uses.
Do not send the cases to the supplier in advance. If they can tune the tool to your examples, you are back to a demo.
Who sits at the table
Three people are enough, and a fourth tends to slow things down.
- The daily user performs the cases. Choose someone who does this work every week, not the most enthusiastic person on the team.
- The process owner decides, case by case, whether the result is acceptable. This is the person who will answer for the outcome later.
- A note-taker writes down what happened, including workarounds and moments of hesitation. Hesitation is data.
The supplier or the internal developer attends to answer questions and to see problems first hand. They do not touch the keyboard and do not explain how it "should" work.
Running the two hours
First ten minutes. State the rules: the daily user works alone with the tool, questions are allowed, but the supplier answers only what a new colleague could look up in the help or the training material.
Next ninety minutes. Work through the cases one by one. Start each with a timer and stop it when the daily user says the case is finished or gives up. Do not coach. If the user is stuck for more than five minutes, note it and move on to the next case.
Last twenty minutes. Go through the notes together and score the cases. Leave the question "should we buy it" for later; first agree on what happened.
Scoring the results
Use three scores and nothing more elaborate:
| Score | Meaning |
|---|---|
| Done | The user finished the case alone and the result is what the owner expected |
| Done with help | The case was finished, but only after a hint, a workaround or a manual step outside the tool |
| Not done | The case could not be finished, or the result was wrong |
Count each score and also note the time per case. A tool where nine of ten cases are "done" but each takes twice as long as today is a different decision from one where seven are "done" and three need a workaround.
Deciding in advance what returns the tool
Agree on the return rules before the session, in writing, so that nobody negotiates them after the results come in. A reasonable starting set:
- Any "not done" on a case that happens daily is a reason to return the tool.
- More than a quarter of the cases scored "done with help" is a reason to return it, unless the supplier can fix the cause in a stated time.
- A result that is silently wrong, such as a saved record with a missing value and no warning, is always a reason to return it. Visible failures are annoying; silent ones are dangerous.
Adjust the numbers to your context, but keep the structure: a short list of conditions, agreed first, applied without exceptions. If you are building the tool yourself, the same rules tell your own developers what to fix before release.
After the session
Send the supplier a short list: cases, scores and the exact steps that failed. Ask for a fix date for each item and agree on a second, shorter session limited to the failed cases. Keep the original case list; it becomes your regression check for every later update.
If the tool passes, write down the date and the scores. They are a baseline for the first weeks of use, and our guide to the first 30 days after launching a new internal tool shows how to continue from there.
Common mistakes
- Letting the supplier choose the cases. That is a demo with extra steps.
- Testing with the most skilled user. Your average user decides whether the tool is adopted.
- Coaching the user mid-case. You will not coach them in daily work either.
- Arguing about results during the session. Record first, discuss after.
- Skipping the return rules. Without them, every weak result becomes a negotiation.
What to do this week
Collect ten real cases from the last month, mark two as awkward and write down your return rules on one page. Book two hours with the daily user and the process owner. If you are still deciding whether to build or buy at all, read five questions before you decide, and for choosing who tries the tool first, see picking the first user of a new internal tool.
Sources
- Our related articles linked above.
- The session format, scoring and return rules are a general working approach, not a standard; any figures are illustrative starting points to adapt to your own context.