An AI agent does not warn you when its information is out of date. It answers a question about prices, opening hours, delivery terms or a product feature in the same calm tone whether the answer is right or wrong. The customer cannot tell the difference, and often neither can the colleague who reads the answer in a hurry.
For a product lead this is an ordinary maintenance problem, not a mystery. Something the agent relies on has changed, and nobody updated it. This article describes a short weekly routine that finds those cases early. It needs no special tooling, only a list, a set of questions and someone who owns the habit.
Where stale answers come from
It helps to name the usual causes, because each one is checked differently.
The source document changed, the agent's copy did not. The agent reads from an indexed set of files or pages. Someone updates the original, and the indexed copy stays as it was until the next refresh. If the refresh is manual or irregular, the gap grows quietly.
The source itself is outdated. The document the agent reads was never updated. An old terms page, a price list that lives in two places, a policy that was replaced but not removed. The agent is faithful to a source that is wrong.
Two sources disagree. An old and a new version both sit in the knowledge base. The agent picks one, and which one it picks may change from day to day.
The agent answers from general knowledge. When the sources do not cover a question, a model may fill the gap with something plausible from its training. That is not a document that went stale; it is an answer with no current source behind it at all.
A rule changed outside the documents. A discount ended, a service area was reduced, a product was withdrawn, and the change was only announced by email. Nothing the agent reads reflects it.
Most weeks none of these will have happened. The routine exists for the week one of them has.
Step 1: Keep a source list
Write down every source the agent can draw on. For each, record:
- what it is and where it lives;
- who owns it, meaning the person who would know that it changed;
- when it was last confirmed as current, and by whom;
- how it reaches the agent, and when the agent's copy was last refreshed.
This is a short table, not a project. Most agents rely on a handful of sources, and the list often turns up surprises on the first pass: a folder nobody remembers adding, or a document with no owner. If the agent reads company documents, the checklist in preparing company documents for an AI agent covers how to settle which version is the source of truth and who is responsible for it.
The "last confirmed" date is the most useful column. It is not the date the file was edited; it is the date a person looked at it and said it is still right. A document that has not changed for a year may be perfectly current, but only if someone has checked.
Step 2: Ask the owners one question
Once a week, or on a rhythm that matches how fast your facts change, ask each source owner a single question: has anything changed that this source does not yet reflect? A yes or no is enough. A short message to the owner works, and a reply of "no change" is itself a record.
Step 3: Run a fixed set of test questions
Build a set of questions whose correct answers depend on facts that can change. Think of the things customers and colleagues ask about most:
- current prices, fees or discounts;
- opening hours, lead times and delivery conditions;
- what is included in each product or service;
- who to contact for which kind of request;
- the current version of a policy, such as returns or cancellation.
For each question, write the correct answer next to it, taken from the current source and dated. Ten to twenty questions are enough to start. Ask them to the agent every week and compare its answers with your notes.
Two details make this work. First, keep the questions the same from week to week, so a change in the result means something changed in the sources or the agent. Second, update the expected answers whenever a source legitimately changes. If you forget, the test will report a failure that is really a stale answer key.
Add a few questions the sources should not be able to answer. If the agent responds confidently instead of saying it does not know or handing over to a person, that is a separate finding. It means the agent may be filling gaps from general knowledge.
Step 4: Sample real answers
A fixed test set only catches what you thought to ask. Real use is wider. Each week, pull a small sample of recent answers, enough that one person can read them in half an hour, and check each against the current source.
Favour answers that contain a fact likely to change: a number, a date, a condition, a name. An answer that is a polite greeting tells you little. If you already review conversations for quality, this fits naturally into that work; the routine in scoring AI agent conversations with a weekly sample describes how to draw and score a sample, and you can add a "was this fact current?" column to it.
When you find a stale answer, look at what the agent cited or relied on. That tells you which of the causes above you are dealing with, and therefore who needs to act.
Step 5: Fix the source, then retest
When something is out of date, resist the urge to correct the agent's answer directly or add a special instruction that overrides the old fact. That hides the problem and leaves the wrong source in place for the next question.
Instead:
- Update or remove the source, or resolve the conflict between two versions.
- Refresh the agent's copy and confirm that the refresh actually happened.
- Ask the same question again and confirm that the answer now matches.
- Record what was wrong, what changed, who did it and the date.
Step 6: Decide what counts as serious
Not every stale answer carries the same weight. An outdated opening hour on a quiet Sunday is a nuisance. A wrong price, a withdrawn offer presented as available, or an incorrect statement about a legal or contractual condition can cost money or trust.
Agree in advance what happens in the serious cases. For example, the agent is paused on that topic or sends such questions to a person until the source is corrected, and the affected customers are identified and contacted. Having this decided beforehand saves time when it matters. If the agent acts on its own in the affected process, the level of freedom it has may also need a second look; see setting an autonomy level for each process. For how to stop an agent quickly and handle the aftermath, read incident response and kill switches for AI agents.
A one-page record
Keep the routine on a single page that anyone on the team can open:
- the source list, with owner, last-confirmed date and last refresh;
- the test questions with their current expected answers;
- the date of each weekly run and its result;
- the stale answers found, their cause, the fix and the retest;
- the name of the person responsible for running the routine, and a deputy.
The page is also your evidence if a customer or a colleague later asks how you know the agent's information is current.
Weekly checklist
- Each source has an owner, a last-confirmed date and a known refresh route.
- Owners were asked whether anything has changed.
- The fixed test questions were run and compared with dated expected answers.
- A small sample of real answers was checked against current sources.
- Questions the sources cannot answer were tested for a safe response.
- Every stale answer led to a fixed source, a refresh and a retest, not a patch on the answer.
- Serious cases followed the agreed steps.
- The record was updated and the next run is scheduled.
The aim is not to distrust the agent. It is to make freshness a habit with an owner, so that when the world changes, your agent changes with it and you hear about the gap from your own check rather than from a customer.
