When an AI agent disappoints, the first suspect is usually the model. In our experience the more common cause is the brief: a paragraph that says what the author hopes for, but gives nobody a way to tell whether the hope was met.
"Answer customer questions about delivery in a helpful and professional way" is a typical example. It sounds reasonable. But two people reading the same agent reply will disagree on whether it was helpful, and neither can point to a sentence in the brief that settles it.
This article gives a product or operations lead a one-page template. The test of a good brief is simple: a colleague who did not write it can use it to mark a sample of outputs as pass or fail, and mostly agrees with you.
Why vague briefs fail
A vague brief fails in three ways, and they get more expensive in that order.
The agent guesses. Where the brief is silent, the agent fills the gap with something plausible. It picks a tone, invents a policy detail, or answers a question you never meant it to touch.
Nobody can review. A reviewer has nothing to compare the output against, so the review turns into personal taste. The same reply is approved on Monday and rejected on Thursday.
Nothing can be improved. When a result is poor, you cannot say which part of the brief was wrong. You rewrite everything and hope.
A testable brief fixes all three, because every statement in it can be checked against an output.
Part 1: the goal
Write the goal as an outcome for a named user, in one or two sentences, and attach the decision it supports.
Weak: "Help the support team with refunds."
Better: "For each incoming refund request, prepare a draft reply and a recommended decision (approve, reject, ask for more information) so that a support agent can send it after a quick read. The agent does not send anything itself."
The better version says who uses the output, what the output is, and what happens next. It also contains a boundary - nothing is sent - which already belongs to the next part.
Add one line on how success is measured in practice, such as "a support agent accepts the draft with no or minor edits in at least 8 of 10 cases." The number is yours to choose; what matters is that it exists before the agent starts.
Part 2: the limits
Limits are the part people skip, because they feel negative. They are also the part that prevents most incidents. Write three short lists.
Allowed. What the agent may read and which actions it may take. Be specific: "reads the order record and the refund policy page", not "has access to the order system".
Not allowed. What it must never do, even if asked politely by a user. Examples: promise a refund amount, change an order, give legal or medical advice, quote prices that are not in the price list.
Hand over to a person. The situations where the agent stops and passes the case on, and to whom. Typical triggers are a missing source, a request outside the scope, an angry or distressed customer, an amount above a threshold, or the agent's own uncertainty. If you have already decided how much freedom the agent gets in each process, link that decision here; our article on autonomy levels for AI agents covers how to set it.
Limits should also name the sources. The agent answers from a defined set of documents, and says so when the answer is not there. If those documents go out of date, the brief goes out of date with them, which is why a source freshness check is part of running the agent, not an extra.
Part 3: examples of good and bad output
Adjectives such as "concise", "friendly" and "accurate" mean different things to different readers. Examples mean the same thing to everyone. Include at least three.
A good output. A real or realistic case with the input, the output you would accept, and one sentence on why it is good. For instance: the draft states the decision, cites the policy clause, and asks for nothing the customer already provided.
A bad output. A case that looks fine at first glance but fails. The most useful bad examples are the plausible ones: a polite reply that promises a refund the policy does not allow, or a confident answer based on last year's price list. Say why it fails.
A borderline output. A case where reasonable people could disagree, plus your ruling. This is the example that saves the most review time later, because it records a judgment that would otherwise be argued again every week.
Keep the examples short and keep them in the brief itself. If the examples live in someone's head or a chat thread, the next person to maintain the agent will not have them. Avoid examples containing real customer names or personal data; replace them with realistic placeholders.
Part 4: the acceptance test
The acceptance test turns the first three parts into a check you run before the agent goes live, and again after any change.
- Build a small test set. Twenty to thirty real or realistic inputs is enough to start. Include ordinary cases, a few awkward ones, and several that should trigger a handover or a refusal.
- Write the pass criteria per case. Not a perfect answer, but a short list: the decision is correct, the policy is cited, no forbidden promise is made, the right sources were used.
- Run the agent and mark the results. Ideally two people mark independently. Where they disagree, either the criteria or the brief is unclear, and you have found something to fix.
- Set the threshold in advance. For example: at least 90 percent pass overall, and no failures on the "not allowed" cases. A single forbidden promise should fail the whole run, whatever the average says.
- Record the result. Date, version of the brief, model, pass rate, and the failures you decided to accept.
This is not a scientific evaluation, and it does not need to be. It is a cheap, repeatable check that turns "it seems to work" into "it passed 27 of 30 cases and the three failures are known". Public guidance on risk management, such as the NIST AI Risk Management Framework, makes the same point at a larger scale: define what you expect, measure it, and keep measuring.
A one-page template
You can copy this structure into any document tool.
- Name, owner and deputy. Who is responsible for this brief.
- Goal. The outcome, the user, the next step, and the success measure.
- Allowed. Sources the agent may read, actions it may take.
- Not allowed. Hard prohibitions.
- Hand over. Triggers and the person or queue that receives the case.
- Examples. One good, one bad, one borderline, with a reason for each.
- Acceptance test. Test set location, pass criteria, threshold, date of last run.
- Version history. What changed and why.
If it does not fit on one page, the agent is probably doing too many things. Split the work into two agents with two briefs, which is easier to test and easier to hand over later.
Common mistakes
- Describing the process instead of the result. Step-by-step instructions are useful, but without an outcome you cannot tell whether following them was enough.
- Only positive examples. An agent and a reviewer both need to see what failure looks like.
- Prohibitions that are not testable. "Be careful with sensitive topics" cannot be marked pass or fail. "Never state a diagnosis; refer to a doctor" can.
- Testing once. The brief, the model and the sources all change. A result from March says little about October.
- No owner. An unowned brief drifts. When the agent is later retired or moved to a new team, nobody knows what it was meant to do, and the handover to a new team starts from zero.
- Writing the brief and judging the result alone. The author reads the output with the intent in mind and fills the gaps without noticing. A second reader finds them.
Where to start
Pick one agent that already exists, even an informal one, and rewrite its brief on one page using the template. Build a test set of twenty cases from real work. Run it once and look at the failures. In most teams this takes an afternoon and shows exactly where the old brief was silent.
If an agent can read or change anything beyond its text output, review its permissions at the same time; the principles are in our guide to least privilege for AI agents.
A brief that can be tested is also a brief that can be improved, handed over and trusted. That is most of what separates an agent that works in a demo from one that works on an ordinary Tuesday.
