Most pilots end quietly. The agent keeps running, a few enthusiastic people keep using it, and nobody formally decides anything. Six months later it either has far more access than anyone approved, or it is silently ignored and still costing money. A pilot is only useful if it ends in a decision.
This article is for the manager who owns that decision. It assumes you already chose a process, ran the agent in a limited way, and collected some data. If you are earlier in the journey, see choosing the first process, staged rollout and rollback criteria and measuring whether agents really save work. Here the question is what to do with the results.
Start from what you wrote before the pilot
The decision should be a comparison, not a debate. Before launch you should have written down three things: what success looks like, what would make you stop, and who decides. Bring those documents to the review and judge the pilot against them.
If you did not write them, say so, and write them now before looking at the numbers. Setting the bar after seeing the results is how a mediocre pilot gets described as a success. It is better to admit the pilot was loosely defined and extend it with proper criteria than to pretend it was rigorous.
Gather five kinds of evidence
Keep the pack small. A review that needs forty slides usually has no clear answer.
- Value. Compare the unit of work, such as tickets handled or reports drafted, with the baseline you recorded before the pilot. Use the same counting method.
- Quality. How often was the output wrong, and how often did reviewers change it? Separate harmless corrections from ones that would have caused damage had they not been caught.
- Review effort. How long did people spend checking and fixing the agent's work? Include the time of the reviewers, not only the process owner. If review time approaches the time the old process took, the saving is mostly an illusion.
- Incidents and near misses. List every case where the agent acted outside its intended limits, used data it should not have, or did something that needed to be undone. Near misses count, because they show where the next real incident will come from. Your incident response plan should already tell you where these are recorded.
- Cost. Add model usage, platform fees, integration and maintenance time, and review time. Compare it with the value. Our note on cost controls and loop limits explains why usage costs can drift up even when volume stays flat.
Be honest about sample size. Two weeks of data on a few dozen cases supports a cautious next step, not a company-wide rollout.
Four outcomes, not two
Teams often treat the decision as go or no-go. In practice there are four outcomes, and picking the right one matters.
Expand
Choose this when the agent met the success criteria, errors were caught cheaply, there were no serious incidents, and the net value after review and cost is clearly positive. Expansion still needs discipline. Change one dimension at a time: more volume of the same work, or an additional team, or a wider set of permissions. If you widen all three and something goes wrong, you will not know why.
Also re-check the controls before expanding. A review process that worked for fifty cases a week may not work for five hundred. Permissions that were reasonable for a pilot may be too broad once the agent touches more systems. Guidance on excessive agency in the OWASP Top 10 for LLM applications is a useful reminder that more autonomy raises the cost of a single bad input.
Adjust and re-run
Choose this when the idea is sound but the pilot exposed fixable problems: unclear instructions, missing data, a review step that was too slow, or a narrow class of cases where the agent fails. Name the specific change, the new test, and a short date by which you will decide again. "Adjust" without a defined re-test is just postponing the decision.
Hold as is
Choose this when the agent is useful at its current size but you have no evidence to support more. This is a legitimate outcome. Keep the controls, keep measuring, and set a date to review again. Holding is not failure; unlimited expansion for its own sake is.
Retire
Choose this when the value is not there, the review burden cancels the benefit, the risk cannot be brought down to an acceptable level, or the data and contract position is uncertain. Retiring is a good result of a pilot, because the cost of learning this was small. The pilot did its job.
How to retire an agent cleanly
Switching off the interface is not enough. Work through a short list:
- Revoke the agent's credentials, tokens and access to systems.
- Stop scheduled jobs and integrations that could trigger it.
- Decide what happens to the data it collected, logs, and prompts, in line with your data handling and retention rules.
- Check vendor terms for notice periods, exit and data return, as covered in vendor contracts, SLAs and exit.
- Tell the people who used it what replaces it, and restore the previous process.
An agent that is left half-alive, with live credentials and no owner, is a larger risk than one that never existed.
Record the decision
Write a one-page decision note: the outcome, the evidence summary, the criteria you used, what you will do next, the date of the next review, and who made the decision. Include dissent if reviewers disagreed. This protects you later, helps the next pilot start from better criteria, and fits the accountability and continuous-improvement expectations in the NIST AI Risk Management Framework.
A short checklist for the review
- The success and stop criteria written before the pilot are on the table
- Value, quality, review effort, incidents and cost are each summarised on one page
- Sample size is stated, and the conclusion fits it
- One of four outcomes is chosen with a stated reason
- If expanding, only one dimension changes and controls are re-checked
- If retiring, access, data and contracts are closed out
- The decision, owner and next review date are written down
A pilot that ends in a clear, documented decision, including a decision to stop, has done what a pilot is for.
