A few weeks after launch, the feedback starts to arrive. A colleague forwards an answer that was wrong. A customer complains about a tone. Someone notices that the agent updated a record it should not have touched. Each of these is reported as "the agent made a mistake", and the natural reaction is to fix whatever was reported last.
That is a poor way to run an agent. This article is for product and operations leads who receive such reports. It suggests a simple way to sort errors into four types, a way to decide what to fix first, and a routine that keeps the list useful.
Why one list of mistakes does not work
If every problem lands on the same list, the list mixes things that have nothing in common. An outdated price, a rude sentence and a deleted calendar entry are all "errors", but they are caused by different things and fixed by different people. The price is a problem of the source. The sentence is a problem of instructions and examples. The calendar entry is a problem of permissions and approval.
A mixed list also hides risk. The loudest report gets attention, and loudness often follows visibility, not damage. A badly worded greeting is visible to everyone. A wrong action in a back-office system may be noticed by nobody for a month.
So the first step is not to fix anything. It is to decide what kind of error you are looking at.
The four types
1. Wrong fact
The agent states something that is not true: a deadline, a price, a policy, a product detail. Before blaming the agent, check where the statement came from. There are usually three possibilities. The source was wrong or out of date. The source was right but the agent picked the wrong passage. Or there was no source and the agent filled the gap with something plausible.
The fix depends on which one it is. A wrong source goes to its owner. A wrong passage means the documents are hard to tell apart, and the fix is better structure or a narrower set of sources. A gap means the agent should say "I do not know" and pass the case to a person. Our article on stale answers and a source freshness routine covers the most common cause, and the guide to preparing company documents covers the rest.
2. Wrong scope
The agent does something it should not do at all, or answers a question it should have passed on. It gives legal advice, promises a discount, comments on a competitor, or handles a complaint that should go to a person. The facts may even be correct. The problem is that the task was outside what the agent was meant to do.
This is a problem of the brief and the rules. The fix is to state the limits more clearly, add an example of a case to refuse, and check the escalation rules. If you have not written them down, start with when to stop and pass the case and with how to write the brief.
3. Wrong tone
The answer is correct and in scope, but it sounds wrong: too casual for a serious complaint, too stiff for a quick question, too confident when the situation is unclear, or apologetic in a way that implies fault the company has not accepted.
Tone matters, especially in customer communication, but it is the easiest type to fix and the least likely to cause lasting damage. A few good and bad examples in the instructions usually help. The exception is tone that creates a commitment, such as "we will make sure this never happens again". That is a scope problem in disguise, and it belongs in the second group.
4. Wrong tool action
The agent used a connected system incorrectly: it updated the wrong record, sent a message to the wrong person, created a duplicate, or did something twice. These errors are the least visible and often the most expensive, because they change real data.
The causes are typically too broad permissions, missing confirmation for irreversible steps, or unclear instructions about which system is the source of truth. The fixes are technical and organisational at once: narrower access, an approval step, and an activity log that makes the action visible. See least privilege for AI agents and what to record in the activity log.
What to record for each error
Keep the record small. Five fields are enough:
- what the agent did or said, as a copy and not a summary,
- the type, one of the four,
- the cause in one sentence, for example "price list from last year",
- the impact: who was affected and what it cost, even roughly,
- whether it could be undone and how.
The cause field is where the learning is. After a month you will see that most errors in one type share one or two causes. That is far more useful than a long list of single cases.
Deciding the order
Once errors are sorted, you need an order. Three questions work well, asked in this sequence.
First: what was the impact? Did it cost money, expose data, break a promise, or mislead a customer on something that matters? Or did it cost a minute of somebody's time? Impact beats everything else. Any error that touched personal data, payments or legal commitments goes to the top and may justify pausing that part of the agent until it is fixed. The incident response guide describes how to do that without panic.
Second: how often does it happen? Among errors of similar impact, the frequent one goes first. One cause that produces thirty small errors a week is worth more than one that produced a single moderate error. This is where the cause field pays off, because you can count by cause and not by symptom.
Third: can it be undone? A wrong draft that a person reviews before sending is reversible. An email already sent, a record already overwritten, or an order already placed is not. When two errors look similar, fix the irreversible one first.
In practice, this ordering puts the types roughly like this: wrong tool actions and wrong scope near the top, wrong facts in the middle, depending on how they are used, and wrong tone at the bottom. It is a tendency and not a rule. A wrong fact in a contract summary can outrank a harmless tool glitch. Use the three questions and not the type alone.
Where the fix usually lives
When people hear "the agent is wrong", they look for a better model or a cleverer instruction. In our experience that is rarely the first place to look. The fix is more often one of these:
- a source document corrected or removed,
- a rule added to the brief, with an example,
- a permission narrowed or a confirmation step added,
- an escalation threshold lowered, so the agent passes the case earlier,
- a person assigned to own a document that nobody owned.
Changing the model or the prompt is sometimes right, but do it after the cheaper options, and change one thing at a time so you can see the effect.
Do not fix one case at a time
A single reported error tempts you to add a single special rule: "never mention product X to customers from country Y". Ten such rules later the instructions are long, contradictory and nobody knows why they are there.
Before adding a rule, ask whether the case is an example of a type of error. If it is, fix the cause. If it is a real one-off, a short note in the log is enough. Keep a small test set of past errors, and run it after each change, so a fix for one problem does not quietly bring back another. The value scorecard shows how to track quality and rework alongside the rest.
A weekly routine
You do not need a big process. A small, steady one works better:
- Once a week, take a sample of the agent's work, including cases that nobody complained about. Complaints show only part of the picture.
- Classify each error found, using the same four types and the same five fields.
- Count by type and by cause, and pick the top one or two items using the three questions.
- Assign each to an owner with a date. A source goes to its owner, a rule to the person who maintains the brief, a permission to whoever manages access.
- Next week, check that the fix worked, and add the case to the test set.
Keep the types stable. If you rename categories every month, you cannot compare weeks and cannot say whether the agent is improving.
Common mistakes
- Fixing the most visible error first. Visibility is not impact.
- Treating tone as the main problem. It is easy to discuss, because everyone has an opinion, but it is rarely the costliest.
- Blaming the model for a source problem. Check the source before changing anything else.
- Collecting errors without a cause. Without a cause the list cannot guide a decision.
- Having no owner. An error that belongs to nobody will return.
- Reviewing only complaints. The errors nobody noticed are often the serious ones.
Where to start
Take the last ten errors you know about, from complaints, from colleagues or from the log. Give each one a type, a cause and an impact. Look at the result: you will most likely find that two or three causes explain most of the list, and that the item you were going to fix first is not the one at the top. That is the point of sorting; the effort is small and the order of work changes.
