Sooner or later a pilot goes wrong. An assistant sends a customer a wrong price. A classifier files a batch of invoices under the wrong supplier. A summary tool leaves out the one clause that mattered. In most cases nobody is hurt badly, but the next few days decide whether the company learns something or quietly drops AI for a year.
This article describes a simple way to handle that moment: stop quickly, write one page, fix the cause and restart on written conditions. It suits a company of ten to a few hundred people running its first pilots. It is a working practice, not a legal or regulatory procedure.
Why you need this before it happens
Under stress people improvise. The pilot owner wants to keep going because they believe in the project. A manager wants to shut everything down. Staff stay silent because they are not sure whether the problem is "big enough" to report. A short plan agreed in advance removes those arguments.
Three things should exist before the pilot starts:
- A stop rule: a plain statement of what makes the pilot pause.
- A named decider: one person who can pause it without asking anyone.
- A reporting route: where and how anyone, not only the pilot team, can raise a concern.
If you already run an AI agent with an emergency stop, the same thinking applies; see AI agent incident response and kill switches for the technical side.
The stop rule
A stop rule is a short list of triggers. Keep it to what a non-technical person can recognise. Examples:
- A wrong output reached a customer, supplier or authority.
- Personal, confidential or customer data went somewhere it should not have.
- The tool made or triggered a change (a payment, an order, a record update) that nobody approved.
- Error rates in the weekly check rose above the level you set at the start.
- Two or more staff say they no longer trust the output.
One trigger is enough. Do not require agreement that the problem is serious; that debate happens after the pause. Pausing a pilot for two days costs little. Letting a faulty one run for two more weeks can cost a customer.
In practice, "stop" should mean something concrete: switch off the automation, move the process back to the manual route, and tell the people who depend on the output.
Who decides
Name one decider and one backup. The decider should be someone with enough authority to stop work, ideally not the person whose project it is. In a small company this is often the operations lead or the managing director. The pilot owner can always request a stop, but the right to pause should not depend on the owner's agreement.
Also decide who can restart. A good rule is that the person who restarts is not the person who built the pilot.
The one-page incident note
Write the note within one working day of the stop, while memory is fresh. One page is a feature: it forces you to leave out speculation. Use these headings.
- What happened. Two or three sentences in plain language, with date and time. "On Tuesday the assistant quoted the old price list to three customers."
- What it affected. Who or what received wrong output, how many cases, and whether any data left the company. If you do not know yet, say "not yet known" and name who is checking.
- How it was noticed. A staff member, a customer, a weekly check? This tells you which of your controls work.
- What we did. The stop, the switch to the manual route, the people informed.
- What we know and what we do not. Keep facts and guesses apart. A guess about the cause is allowed if it is labelled as a guess.
- Owner and next update. One name and one date.
Do not name individuals as the cause. People make mistakes because the process allowed it; the note should describe the process. If you want a longer format for the later review, see the one-page post-incident review for internal systems.
Find the cause and fix it
Resist the quick patch. Changing a prompt so the specific wrong answer no longer appears fixes one symptom. Ask instead:
- Was the input wrong? Old price lists, duplicated records and unclear documents cause many failures that look like model errors.
- Was the task too wide? A tool asked to do five things will fail on the fifth. Narrow the scope.
- Was there a check? If a person was meant to review the output, did they have the time and the information to do it?
- Was the risk known? If the failure was predictable, why was it not in the pilot plan?
Then choose a fix that addresses the cause, name the person responsible and set a date. Test it on a small sample, ideally in the shadow mode where the tool produces answers but people still do the work. For how to run that stage, see staged rollout with shadow mode and rollback criteria.
Restart conditions
Do not restart on a feeling that things are better. Write the conditions down and tick them off. A useful set:
- The cause is described in the incident note and the fix is in place.
- The fix was tested on a defined sample, for example fifty recent cases, with the result recorded.
- The stop rule was reviewed and, if the incident was not caught by it, changed.
- The people affected by the incident were told what happened and what changed.
- The restart starts smaller than the original: fewer users, fewer case types or a human check on every output for the first weeks.
- The restart decider, who is not the pilot owner, has signed off in writing.
If a condition cannot be met, you have a choice: keep the pilot paused, narrow it further or end it. Ending a pilot with a clear reason is a legitimate result. For a method of choosing between those options, see reviewing a pilot: expand, adjust or retire.
Talking about it
Tell the team what happened, briefly and honestly, even if the damage was small. Staff who see an incident handled calmly and without blame are more likely to report the next one early. Tell affected customers or partners what went wrong and what you changed, in a few sentences, and without technical excuses. Whether you must notify an authority depends on the data involved and your local law; ask your data protection contact if personal data was affected.
Common mistakes
- Debating before stopping. Pause first, then discuss.
- Letting the pilot owner decide alone. They are too close to the project.
- Writing a long report nobody reads. One page, one owner, one date.
- Patching the symptom. The same failure returns in a different form.
- Restarting at full scale. Restart smaller than before.
- Hiding a minor incident. Small incidents reported early are the cheapest lessons you will get.
Summary
A pilot that fails is not unusual; a pilot that fails without a plan is expensive. Agree a stop rule and a decider before you start, write a one-page note without blame, fix the cause and test it on a small sample, and restart only when written conditions are met and someone other than the owner agrees. Setting this up takes an hour. Using it saves weeks.
Sources
- Common practice for handling incidents in early AI pilots, offered as a practical guide. No client results or market figures are claimed in this article. Adapt the triggers, the decider and the restart conditions to your own company, and follow your company rules and local law on personal data and data protection.