AI pilots rarely fail loudly. They fade: the first weeks are exciting, then nobody checks the results, a few errors go unnoticed, and three months later someone asks whether the pilot is worth continuing and nobody can answer. A one-page weekly report prevents that. It takes about fifteen minutes to write and gives you the facts for every decision about the pilot.
Who writes it and who reads it
Name one person who writes the report, ideally the pilot owner, and one person who reads it and can decide: a manager, the business owner or the head of the team that uses the tool. If the reader changes, say so in the report. A report that nobody is expected to read soon stops being written.
Send it on the same day each week, for example Friday at noon, and keep every report in one shared folder. After six weeks you have a record you can compare, which is more useful than any single summary.
Part 1: four numbers
Pick four numbers in the first week and keep them for the whole pilot. Changing them every week makes comparison impossible. A reasonable set for most pilots:
- Volume: how many tasks the AI handled this week (emails drafted, tickets sorted, documents summarised).
- Accepted as is: how many of those a person used without changes.
- Corrected or rejected: how many needed edits or were thrown away.
- Time per task: the average minutes a person spent per task, including checking the AI output.
Take the numbers from a simple log or a shared sheet, not from memory. If the tool cannot give you counts, a person can tally a sample of twenty items each week. A small honest sample beats a large guess. Before the pilot, record the same figures for the manual process so that you have a baseline, as described in how to review a pilot and decide to expand, adjust or retire it.
Part 2: what failed
Counts hide the useful information. Under the numbers, list up to three failures from the week, each with one sentence about what happened and one real example. For instance: "The tool answered a delivery question with last year's terms. Example: ticket from Tuesday." Mark each as one of:
- Wrong content: the answer was incorrect or out of date.
- Wrong tone or format: correct but unusable without rewriting.
- Missed case: the tool should have handled it and did not.
- Should not have acted: the tool did something a person should have decided.
The last category matters most. If it appears even once, note it at the top of the report, because it points to a rule that needs tightening. If you are collecting such cases already, a test set of real cases is a good place to add them.
Part 3: what it cost
Write down the real cost for the week in three lines: the subscription or usage charge, the staff time spent on checking and correcting, and any one-off work such as setup or training. Staff time is the figure most often left out, and it is often the largest. Use a simple hourly figure that your finance person agrees to, and use it every week.
Then add one sentence comparing cost with the baseline: "This week the tool cost about the same as the manual process and saved two hours of waiting time." If you cannot say that honestly yet, write "not yet clear". That is an acceptable answer in the first weeks and a warning sign by week six.
Part 4: the decision for next week
Every report ends with one decision, not a list of wishes. Examples:
- Keep the pilot as it is for another week.
- Change one thing, such as the instructions for refund questions, and measure again.
- Widen the pilot to one more team or one more type of request.
- Pause it until a named problem is fixed.
- Stop, with a short reason.
Change one thing at a time. If you change the instructions, the data and the user group in the same week, you will not know what made the difference. The reader replies with "agreed" or a different decision; that reply is the record that someone is steering the pilot.
A template you can copy
Keep the page to this shape:
- Week and writer / reader
- Numbers: volume, accepted as is, corrected or rejected, minutes per task, each with last week's value
- Failures: up to three, each with an example and a category
- Cost: subscription or usage, staff time, one-off work
- Decision for next week and the reader's reply
Common mistakes
- Changing the numbers every week. You lose the trend.
- Reporting only successes. The report becomes marketing, and nobody trusts it.
- Leaving out staff time. The pilot looks cheaper than it is.
- No decision at the end. The report becomes a diary.
- No reader. The writing stops after a month.
Summary
A weekly one-page report with four fixed numbers, a short list of failures with examples, an honest cost line and one decision gives a small team control of an AI pilot without extra tools. After a few weeks it also answers the question that always comes: should we keep going, change course or stop?
Sources
- Common practice for small teams, offered as a practical guide. No client results or market figures are claimed in this article. Adjust the numbers, the cost method and the reporting day to your own pilot.