Sooner or later someone senior asks the question: "What exactly are our AI agents doing, and should we be worried?" If you have only an answer in conversation, you will give a different one each time. If you have a one-page report, you give the same calm answer every quarter, and you notice problems earlier.
This article describes such a report for a leadership team or a board of a small or mid-sized company. It is not legal advice, and it does not describe what any regulation requires of you. It is a practical format you can adapt.
What the board actually needs
Directors and senior managers are rarely technical, and they do not have time. They usually need to answer three questions:
- Is it working? Is the agent doing something useful, and how do we know?
- Is it safe enough? What could go wrong, what has gone wrong, and who is watching?
- What is expected of us? Is there a budget, a policy or a risk acceptance that only leadership can decide?
Everything else, such as prompt versions, model names or token counts, belongs in the team's own working documents. If a detail does not help answer one of these three questions, leave it out.
The one-page structure
Keep the same six blocks every time. A fixed structure matters more than clever wording, because the reader learns where to look and sees what has changed.
1. Scope: what the agents do
Two or three lines. Which agents are live, which process each one handles, and whether it acts alone or a person approves its output. Add anything new since the last report: a new agent, a new data source, a new permission. If nothing changed, say "no change".
2. Results: what we get
Pick a few measures that were agreed before launch, and show them against the starting point. For example: time per case before and after, share of cases handled without rework, customer response time, and cost per case. Our article on the value scorecard explains how to choose them, and the one on measuring value with a baseline explains why rework and review time must be counted.
Report ranges and honest caveats. "Average handling time fell from 14 to 9 minutes in the first eight weeks; the sample is small and includes one quiet month" is more useful than "40% faster".
3. Incidents and near misses
List what went wrong since the last report, even if it was small: a wrong answer that reached a customer, an agent stopped by a person, a vendor outage. For each, one line on what happened, what it cost, and what changed afterwards. If there were none, say so, but also say how you would know. An agent without an activity log that reports "no incidents" may only be reporting that nobody looked.
Near misses are worth including. A board that sees a near miss with a fix learns that the control works. A board that learns about the same problem from a customer learns the opposite.
4. Open risks
Three to five risks, no more, in plain words. For each: what could happen, how likely it seems, how bad it would be, what is being done and who owns it. A simple High, Medium or Low is enough. A typical list for a company with a customer-facing agent might be:
- the agent gives a wrong answer that a customer relies on;
- customer data is handled in a way the contract or the customer notice does not allow;
- the agent has more access than its task needs, see least-privilege access;
- the company depends on one vendor whose terms or prices may change, see vendor contracts, SLAs and exit;
- the person who understands the agent leaves.
Do not hide a risk by describing it in technical language. If the board cannot tell whether a risk is serious, the block has failed.
5. Cost
Total cost for the period against the budget: subscriptions, usage charges, and the staff time spent on review and maintenance. Include the part that is easy to forget, the time of the people who check the agent's work. If the figure is an estimate, say so. The article on total cost of ownership lists the cost items to count.
6. Decisions needed
This is the block that makes the report useful. Write each request as a decision with options and a recommendation, for example:
- "Approve a pilot of the same agent for the returns team, with a two-month review and a budget of X."
- "Accept the residual risk that wrong answers may occasionally reach customers, given that a person reviews every answer above a set value."
- "Choose between keeping the current vendor and moving to another one before the contract renews on a given date."
If you need no decision this time, write "none". Do not invent one to look busy.
What to leave out
- Activity counts without meaning. "The agent processed 12,000 requests" says nothing about value or risk.
- Jargon. Explain a term in one sentence or avoid it.
- Reassurance. "We take security very seriously" is not information. A named owner and a tested stop procedure are.
- A claim of compliance you cannot show. If you are not certified to a standard or have not been audited against a framework, do not imply it. Saying that you use a framework such as the NIST AI Risk Management Framework as a reference is fine if it is true.
- Blame. Describe what happened and what was changed.
Rhythm and ownership
For most small and mid-sized companies a quarterly report is enough, with a short monthly check inside the team, as described in the monthly review. The exception is a serious incident: customer data exposed, a significant financial error, or an agent stopped for safety reasons. Then leadership should hear within days, not at the next scheduled meeting.
Give the report one owner, usually the person accountable for the agent's process, not the person who built it. The owner collects the figures, writes the page, and answers questions. The report should be reviewed by someone who did not write it before it goes up, for the same reason that any other important output is.
Keep the previous reports. After a year you will have a short, honest history of what the agents did, what went wrong and what was decided, which is what a board, an auditor or an insurer will eventually ask to see. Our article on AI and insurance describes one situation where such a record is useful.
A short template
You can paste this into a document and fill it in:
- Period and owner
- Scope: agents live, changes since last report
- Results: 3 to 4 measures, with starting point and caveats
- Incidents and near misses: what happened, what changed
- Open risks: up to five, with owner and status
- Cost: total versus budget, including review time
- Decisions needed: with options and a recommendation
- What we do not know yet: and when we expect to know
Conclusion
A good board report on AI agents is short, regular and honest. It shows results next to risks, uses the same structure every time and ends with what leadership is asked to decide. It does not try to impress. Its purpose is that the people accountable for the company can see what the agents are doing and can say yes, no or not yet with their eyes open.
