Most small companies do not need a fleet of agents. They need one repetitive, annoying job done faster, with someone clearly responsible for the result. A pilot is how you find out whether an AI agent can do that job in your business, with your data, at a cost that makes sense, without putting customers or money at risk.
This guide is for the person who has to decide: an owner, an operations lead, or a manager who has been asked, "Should we try an AI agent?" It covers what to pick, how to limit it, and how to decide at the end.
Start by asking if you need an agent at all
The word "agent" is used loosely. Anthropic draws a useful line: workflows run models and tools through predefined steps, while agents let the model decide its own steps and tool use. Its advice is to find the simplest solution that works and add complexity only when needed, because agents often trade higher cost and latency for flexibility.
For a small company that means a simple order of preference:
- A single model call with good instructions, such as drafting a reply or summarizing a document.
- A fixed workflow, where the steps are always the same and the model handles one or two of them, for example sorting incoming email into categories and filling a template.
- An agent, only when the steps genuinely cannot be predicted in advance and the work needs judgment across several tools.
If option 1 or 2 solves your problem, you will get a cheaper, more predictable result. That is a success, not a smaller ambition. Our guide on how to choose your first AI use case goes deeper on the selection step.
Choose the process: four tests
Use a short filter. A good first process passes all four.
| Test | What to look for | Warning sign |
|---|---|---|
| Repetitive | Happens many times a week with a similar shape | Happens twice a year |
| Measurable | You can count time, errors, or turnaround today | "It will just feel better" |
| Contained | A wrong result is caught and fixed before it reaches a customer or a payment | One mistake means a refund, a legal issue, or a lost client |
| Owned | One named person cares about the result and will review it | "Everyone" owns it |
OpenAI's practical guide to building agents points to similar signals: workflows with complex decisions, rules that have become hard to maintain, or heavy reliance on unstructured data such as emails and documents. Those are places where an agent can add something a fixed script cannot.
Good first candidates are internal and reviewable: triaging a shared inbox, preparing a first-draft answer to common support questions, extracting fields from supplier invoices for a person to approve, or compiling a weekly status summary from several tools. Poor first candidates are anything that moves money, signs contracts, or sends messages to customers without review.
Give the agent a small job description
Write a one-page brief before any tool is set up. It should answer:
- Task. What exactly does the agent do, and what does it never do?
- Inputs. Which data and systems can it read? Start with the minimum.
- Outputs. What does it produce, and who receives it?
- Human step. Where does a person review or approve?
- Owner. Who is responsible for the pilot day to day?
Keep permissions narrow. OWASP's guidance on excessive agency names three typical causes of trouble: too much functionality, too many permissions, and too much autonomy. For a pilot, the answer to each is the same: give the agent only the tools, access, and freedom the task needs. A read-only connection to the inbox is a safer start than full access to send and delete. For the access side, see our guide on identity and least privilege for AI agents.
Set the limits before you switch it on
Limits are cheap to add at the start and awkward to add after something goes wrong. For a four-week pilot, decide these in advance:
- Spend. A daily and a weekly ceiling, plus an alert before you reach it. Our article on AI agent cost controls explains the types of limits that work.
- Run limits. A maximum number of steps, retries, and minutes per task. Anthropic notes that agent tasks often include stopping conditions, such as a maximum number of iterations, to keep control.
- Human handoff. OpenAI recommends planning for human intervention in two cases: when the agent exceeds a failure threshold, such as repeated retries, and when an action is high-risk or irreversible. Write both rules down.
- A stop switch. One person who can turn the agent off in a minute, and knows how. See kill switches and incident response for AI agents.
- Data boundary. A list of what the agent must not see, such as personal data you have no need to process or confidential client files.
Define success and stop numbers
This is the step most pilots skip. Without numbers, the end of a pilot becomes an argument about opinions.
Before the start, record a baseline for two or three weeks of normal work, or at least a sample: how long the task takes, how many items per week, how often errors occur. Then set:
- A success line. For example, "median handling time falls by a third with no increase in corrections."
- A quality line. For example, "at least 9 out of 10 reviewed outputs are usable with no more than a small edit."
- A stop line. For example, "any output that reaches a customer without review, or spend above the weekly ceiling twice, ends the pilot."
The numbers above are examples; choose ones that fit your process. What matters is that they are written down and agreed by the owner before day one. To check quality properly, build a small set of real past cases with known good answers and run the agent against them first. Our guide to evaluating AI agent reliability before production shows how.
Run the pilot in four weeks
A simple schedule keeps it honest:
- Week 0, preparation. Brief, baseline, limits, test cases, access. Nothing runs on real work yet.
- Week 1, shadow mode. The agent works on real inputs, but its output is only compared with what the person does. Nothing is sent or changed.
- Week 2, supervised. A person reviews every output and uses it when it is good. Log every correction and why.
- Week 3, supervised with less review. If quality is on target, review a sample instead of everything. Keep high-risk cases fully reviewed.
- Week 4, measure. Compare with the baseline, add up total cost, including the time spent reviewing, and collect the owner's and users' feedback.
Keep a simple log: number of tasks, number accepted as is, edited, rejected, cost, and any incident. A spreadsheet is enough. The review time matters because a pilot that saves ten minutes of work but adds eight minutes of checking has not saved much.
Decide at the end: stop, fix, or expand
At the end of week 4, make one of three decisions and record the reason.
- Stop. The result missed the success line, quality was too low, or the cost was higher than the value. This is a normal, useful outcome. You spent a few weeks and a small budget instead of a large project. Write down what you learned.
- Fix and repeat. The result was close, and you can name the specific problem, such as unclear instructions, missing data, or one failing case type. Change one thing and run another short cycle.
- Expand one step. The agent hit the targets with no serious incident. Expand along one dimension only: more volume, a second similar process, or a little more autonomy. Do not do all three at once.
If you expand, move from "pilot" to "owned service": a named owner, a regular review of cost and quality, and a place where users can report problems. NIST's AI Risk Management Framework describes this ongoing discipline in four functions: govern, map, measure, and manage. You do not need a large programme to apply the idea; a short monthly review covering what the agent does, how it performs, and what could go wrong is a good start. For the bigger picture of pilots that grow, see how AI pilots scale.
Before you start
If you want an outside view on choosing the process or setting up the pilot, talk to us. We can help you scope it so the first result is clear in weeks, not months.