Most teams check an AI agent closely in the first week and then rarely look at it again until someone complains. The monthly review is useful, but a month is a long time for a broken data source or a loop that quietly burns budget. A daily check closes that gap. It is not a monitoring project. It is a short routine: eight numbers, one reader, a few agreed actions.
This article is about that routine. If you are deciding what data you may collect at all, start with our article on observing AI agents in production without leaking data. For the longer monthly view, see the monthly review of benefits and costs.
Who reads it
One named person reads the numbers each working day, usually the product or process owner of the agent, with a named deputy. It should take about two minutes. If it takes longer, there are too many numbers or they are scattered over several tools. The reader does not fix problems alone: the action column below says whom to call.
The eight signals
For each signal: what it tells you, and what the reader does when it moves.
1. Cases handled
How many cases the agent took today. A sudden drop usually means something upstream broke (a form, a mailbox, a connection). A sudden rise may mean a campaign, or a loop. Action: compare with the same weekday; if far outside the usual range, check the intake first.
2. Cases passed to a person
The share of cases the agent hands to a human. Too low may mean the agent is answering things it should not; too high means it is not saving time. See our article on when the agent should stop and pass the case. Action: read three handed-over cases and decide whether the rule or the agent is at fault.
3. Corrections by staff
How many outputs a person changed or rejected before they were used. This is the most honest quality signal you have every day, because it needs no extra review work. Action: sort the corrections by type using the groups in classifying AI agent errors.
4. Time to answer
The median time and the slowest tenth of cases. The median shows normal service; the slow tail shows where customers wait. Action: if the tail grows while the median stays flat, look for one failing tool or system, not for a general slowdown.
5. Cost of the day
Spend on usage, compared with the daily share of the monthly budget. Action: if the day is above the agreed limit, check for loops and repeated calls before anything else. Our guide to budgets and loop limits describes the limits to set in advance.
6. Failed or repeated tool calls
How many times the agent tried an action in a connected system and failed, or tried the same action again and again. A small number is normal. A sharp rise often comes before visible errors. Action: pass the example to whoever owns the connected system.
7. Answers from old sources
Whether the documents the agent relies on are still current, and whether any answer today used a source past its review date. Action: tell the document owner. The routine in checking source freshness fits here.
8. Complaints and negative feedback
Count of complaints, thumbs-down or "this was wrong" notes that mention the agent. Small numbers, read one by one. Action: each complaint gets an owner the same day, and a pattern of two or three similar ones becomes a ticket.
How to set thresholds
Do not borrow thresholds from the internet. They depend on your volume, your processes and your tolerance.
- Collect two weeks without alerts. Write down each signal once a day.
- Define the normal range for each: the lowest and highest value you saw on ordinary days.
- Define the action point: the value at which someone must act the same day. A practical start is a margin beyond the normal range, then adjust after a month.
- Write the action next to it: who is called and what they check first.
Worked example (invented numbers, for illustration only). Suppose that in two weeks of ordinary working days the agent handled between 80 and 120 cases a day, and passed between 12 and 24 of them to a person. The handover share then ranged from 12/120 = 10% to 24/80 = 30%, a wide spread. Looking only at weekdays shows that Mondays are the busy days with the lower share, and Fridays the quiet days with the higher one. So the team sets the normal range for handovers at 10–30% and the action point at 40%, and checks Fridays against Fridays. On a day with 90 cases and 36 handovers the share is 36/90 = 40%, so the reader reads three handed-over cases. If they show the same missing document, the fix is a document, not a new prompt. Daily cost works the same way: with a monthly budget of 3,000 and 20 working days, the daily share is 3,000 / 20 = 150, so a day at 150 is on plan and a day at 300 is double and needs a look the same day.
Keeping the check short
- One screen, one reader. If you need three tools, build a single table first, even a spreadsheet filled by a short script.
- Show the trend, not only today. Add the previous four same weekdays next to each number.
- Mark the owner of each signal. A red number without an owner is ignored by the third day.
- Log what you did. One line a day: "Handovers 40%, read three, missing price list, told the document owner." That line is also what you show in the monthly review. See what to record in the activity log.
Common mistakes
- Too many numbers. Twenty charts mean none is read. Eight is already enough.
- Averages only. An average hides the few customers who waited very long.
- Thresholds set once and never changed. Volume and processes change; review the thresholds monthly.
- No action written down. The reader then guesses, and the guess depends on who is on duty.
- Treating a quiet day as a good day. A drop in cases can be a broken intake, not success.
Where to start this week
Pick the eight signals from your own tools, even if some start as manual counts. Name the reader and deputy. Collect two weeks of numbers without any alerts, then set the normal range and the action point for each, and write the action beside it. After a month, remove any signal that nobody acted on and add one only if a real problem would have been caught by it.
Sources
- Your own agent logs, case counts, staff corrections and billing data, as collected in the routine described; all figures in the worked example are invented for illustration.
- Our related articles linked above.