Your team has built an agent, run it against a set of test cases, and the results look good. The decision in front of you is whether to let it work on real requests. That is a different question from "does it pass the tests?" Test cases are chosen by people who already know what the agent is supposed to do. Real users bring odd phrasing, missing data, half-finished requests, and situations nobody listed.
This article is for the manager who has to approve that step. It covers what happens after evaluation: how to expose the agent to reality gradually, what to measure, and when to pull it back. How to build the test set itself is covered in evaluating AI agent reliability before production. The tracing you need during a pilot is in observing AI agents in production without leaking data.
Why a test score is not a release decision
Evaluation answers "how does the agent behave on cases we thought of?" A rollout answers "how does it behave on cases we did not think of, with real consequences?" Three gaps usually open up:
- Distribution gap. Real requests are messier and more varied than a curated set.
- Environment gap. Live systems have slow responses, stale records, permission quirks, and rate limits that a test sandbox hides.
- Human gap. People react to the agent. They rephrase, escalate, ignore its output, or rely on it more than they should.
None of these shows up in a score. They show up only when the agent meets real work, so the safest way to meet them is in small, reversible steps.
Stage 1: shadow mode
In shadow mode the agent receives real inputs, produces its answer or proposed action, and nothing is sent or changed. A person continues to do the work as before. Someone then compares the agent's output with what the person actually did.
Shadow mode is cheap and carries almost no customer risk, and it gives you three things:
- A realistic sample of requests to add to your regression set.
- A first measure of agreement with your best people, including where the humans disagree with each other.
- Early evidence of operational problems: timeouts, tool errors, cost per task.
Run it long enough to see the ordinary mix of work, including the slow end of the week or month. A few days of a quiet queue tells you little. Set the exit rule in advance, for example "agreement with the reviewer on at least the agreed share of cases, and no critical errors in the sample". The numbers depend on the task and the cost of a mistake; the point is that they exist before you look at the results.
Stage 2: a limited pilot with a human in the loop
Next, let the agent act, but in a narrow slice. Limit it along one or two dimensions:
- Scope: one request type, one product line, or one region.
- Volume: a small percentage of traffic, or a fixed number of tasks per day.
- Authority: the agent drafts and a person approves, or the agent acts only on low-value cases.
- Audience: internal users or a friendly customer group first.
This is the same idea software teams call a canary release: expose a small part of the workload to the new version, compare it with the rest, and widen only if the comparison holds up. The Google SRE Workbook describes the approach for software releases, and the logic carries over to agents. A related tool is the feature flag, a switch that turns a capability on or off without a new release; Martin Fowler's overview is a good short introduction.
Keep a comparison group where you can. If similar tasks are still handled the old way, you can compare resolution time, error rate, and customer satisfaction against a real baseline rather than a feeling.
Make approval honest. If a reviewer approves every draft in two seconds, the human check has become decoration. Sample the approved items later and look for mistakes that slipped through.
Decide the thresholds before you start
The most common failure in rollouts is not a bad agent but an undefined success. When nothing was agreed in advance, every result can be argued either way. Write a one-page rollout charter that names:
- The owner. One person who can pause the agent without asking anyone.
- Go-live thresholds. The levels of task success, escalation rate, response time, and cost per task that allow moving to the next stage.
- Hard limits. Events that are never acceptable, such as exposing customer data, taking an action outside the agent's permissions, or giving a wrong answer on a regulated topic.
- Review rhythm. When the group looks at the numbers, for example daily in week one, then weekly.
NIST's AI Risk Management Framework organises this kind of work around govern, map, measure, and manage. A rollout charter is a small, practical version of the "govern" and "manage" parts: someone is accountable, and there is a plan for what happens when measurements go the wrong way.
Rollback criteria: agree when to switch it off
Pulling an agent back feels like failure unless you decided to do it in advance. Then it is just the plan working. Define triggers in two groups:
Stop immediately (the owner acts, review comes after):
- Any breach of a hard limit.
- A pattern of the same serious error, even if the overall score still looks fine.
- Cost per task or call volume running far above the budget.
- A failure of the monitoring itself, because you can no longer see what the agent does.
Pause and review (the group decides within a set time):
- Task success falling below the threshold for a defined stretch.
- A rising share of customer complaints or manual corrections.
- A change in the underlying model, tools, or data that the agent has not been re-tested against.
Make the stop a single step: a flag, a permission revoked, a routing rule that sends work back to the old process. Test it before the pilot, not during the first incident. Cost caps and loop limits belong in the same plan; see cost controls, budgets, and loop limits for AI agents. For the incident side, including who does what in the first hour, see AI agent incident response and kill switches.
Stage 3: widen in steps and keep measuring
If the pilot meets its thresholds, widen one dimension at a time: more volume, then more request types, then more authority. Do not change all three at once, or you will not know which change caused a new problem.
Keep three habits after the full release:
- Feed failures back. Every corrected or escalated case goes into the regression set, so the same mistake is caught next time.
- Re-test on change. A new model version, new tool, or new data source is a new release and goes through at least a short shadow period.
- Keep a human path. Customers and staff should be able to reach a person without fighting the agent.
If your organisation has not yet run a first agent project, the practical starting point is described in a first AI agent pilot: a small business playbook.
A short checklist for the approver
Before you say yes to going live, ask the team:
- Which stage are we in, and what exactly did the last stage prove?
- Where are the written thresholds, and who agreed them?
- Who can switch the agent off, and when was that last tried?
- What is the comparison baseline?
- What does the agent still not handle, and where do those cases go?
If any answer is "we will decide that when it happens", the rollout is not ready. The agent may be fine; the plan around it is not.