An AI agent performance review is a monthly meeting where the people who own your AI agent look at a month of scored conversations, compare it with the goals, and decide what changes. It works like a human agent’s review: same scorecard, real evidence, clear goals. The difference is the outcome: a change to content, scope or routing, not a coaching plan.
In short
- A monthly AI agent performance review turns scores into decisions: expand the bot, hold it, or take topics away from it.
- It differs from a human review in who attends, what the evidence is and what the consequences are.
- Five roles belong in the room, and QA presents the findings, not the team that deployed the bot.
- A one-page report and a 60-minute agenda keep it short enough to happen every month.
- A written decision rule for expanding, holding or restricting topics stops the review from becoming a debate.
This guide is about the ritual: cadence, people, report and decisions. For which metrics to track, see our guide on how to measure AI agent performance. For the scoring itself, our AI agent QA page shows how Kaizo scores every AI agent conversation on the same scorecard as your team.
Why give an AI agent a performance review?
Give an AI agent a performance review because dashboards report and people decide. Someone has to own the decision to give the bot more work, keep it where it is, or take work away.
Most teams already have the data. The helpdesk counts automated resolutions, the vendor shows intents and drop-offs, and QA scores the conversations. What is often missing is a fixed moment when someone with authority looks at all of it together and makes a call. Without that, a known problem with refund answers can sit in a chat thread for weeks.
AI risk guidance expects that moment too. The NIST AI Risk Management Framework (January 2023) lists, under Manage 4.1, that post-deployment monitoring plans are implemented, including mechanisms for input from users, appeal and override, decommissioning, incident response, recovery and change management. A monthly review is a simple way to give each of those a home.
There is also a fairness reason. Your human agents get reviewed, coached and held to a standard. When the bot handles a growing share of your conversations, holding it to the same discipline is what makes the comparison honest, and it is what your leadership will ask about.
How is an AI agent review different from a human one?
An AI agent review looks at patterns across thousands of conversations and ends in changes to content and configuration. A human review looks at one person’s work and ends in coaching.
| Human agent review | AI agent performance review | |
|---|---|---|
| What is reviewed | A sample or all of one person’s conversations | Every conversation the AI agent handled that month |
| Unit of analysis | The person | The topic, the article, the rule |
| Who attends | The agent and their team lead | QA, the bot owner, the content owner, support ops, the CX leader |
| Typical finding | A skill gap or a habit | An outdated article, a missing rule, a late handover |
| Outcome | Coaching, a goal, a development plan | A content fix, a scope change, a routing or escalation change |
| Consequence of a bad month | A plan to improve | Topics restricted or handed to humans until fixed |
| Fairness concern | Scored for things outside their control | The bot’s errors landing on a human’s score |
One rule from that last row is worth writing into your process: a bot’s error never lands on a human agent’s score. If the AI agent promised a refund and the human agent had to withdraw it, the failed criterion belongs to the bot.
Who should be in the room?
Five roles belong in an AI agent performance review: QA presents, the bot owner and content owner take the fixes, support operations speaks for the queues, and the CX leader decides.
| Role | What they bring | What they leave with |
|---|---|---|
| QA lead (presents) | The scores, the trends, the transcripts behind each finding | Criteria changes for next month |
| AI agent owner | How the bot is configured, what changed this month | Prompt, routing and escalation changes |
| Knowledge content owner | Which articles the bot answers from | Article fixes, with a date |
| Support operations lead | Handover volume, queue impact, repeat contacts | Staffing or routing adjustments |
| CX leader (decides) | Goals and risk appetite | The expand, hold or restrict decisions |
QA presents because the finding should come from the people who did not build the bot. The team that deployed it is also the team whose project looks good when the numbers look good. Our post on who is checking your AI agent explains how to split the finding, the fix and the authority to switch a flow off.
Invite risk or compliance when the month includes a critical failure on a regulated topic. Invite the vendor when you need their help with a fix, not to present the results.
The one-page AI agent performance report
The report fits on one page: a headline number against its goal, the criteria that changed, the topics at risk, the causes behind the failures and the decisions you need.
Copy this template and fill it from your QA tool and helpdesk each month:
| Section | What goes in it | Where it comes from |
|---|---|---|
| 1. Headline | Resolved and correct rate against the goal, this month and last | QA scores plus repeat contacts |
| 2. Volume and scope | AI agent conversations, share of all conversations, topics it handles | Helpdesk |
| 3. Shared criteria | Scores on the criteria humans and the bot share, side by side with the human team | QA scorecard |
| 4. Bot criteria | Pass rates on grounding, policy limits, scope and escalation | QA scorecard |
| 5. Topics at risk | Each topic below its goal, with conversation count | QA scores by topic |
| 6. Critical failures | Count, what happened, customers affected, status | QA findings |
| 7. Top three causes | The article, rule or setting behind most failures, with one transcript each | Root-cause analysis |
| 8. Last month’s fixes | What shipped and whether the scores moved | Re-scored conversations |
| 9. Handover | Late or missing handovers, context passed on | Handoff criteria |
| 10. Decisions needed | Expand, hold or restrict, per topic | The decision rule below |
Keep it to the numbers that lead to a decision. If a row has not changed a decision in three months, drop it from the page and keep it in the appendix.
In Kaizo, the scores for sections 3 to 9 come from the same scorecard. Kaizo Insights consolidates QA, sentiment and coaching into one view that is ready to present or export, and root-cause analysis groups the failures behind a dip.
Worked example: an October review
In this example the AI agent improved overall but failed on one topic, so the review expands one topic, holds three and restricts one. The numbers are an illustration we built, not a customer’s data.
The headline section of the October report:
| Measure | September | October | Goal |
|---|---|---|---|
| AI agent conversations | 11,200 | 12,400 | |
| Resolved and correct | 70.4% | 73.1% | 75% |
| Grounded in our content (pass rate) | 95.8% | 98.0% | 98% |
| Policy limits (pass rate) | 97.2% | 98.1% | 99% |
| Escalated on the right trigger | 91.0% | 89.5% | 93% |
| Repeat contact within 7 days | 6.8% | 6.1% | 5% |
| Critical failures | 4 | 1 | 0 |
The headline moved the right way, by 2.7 points, but escalation went the wrong way. That is the kind of mixed month where a review earns its hour: the overall number would have been reported as good news, while the drop in correct escalation needs an owner.
The topic section is where the decisions come from:
| Topic | Conversations | Grounding and policy pass rate | Critical failures | Decision |
|---|---|---|---|---|
| Order tracking | 4,900 | 99.3% (September 99.1%) | 0 | Expand: add delivery-change requests |
| Returns | 2,600 | 97.8% | 0 | Hold |
| Product questions | 2,700 | 97.1% | 0 | Hold, fix the two articles behind most failures |
| Refunds | 1,300 | 94.6% | 1 | Restrict: hand partial refunds to humans until fixed |
| Account changes | 900 | 98.4% (September 97.6%) | 0 | Hold one more month before expanding |
The meeting ends with five decisions, each with an owner and a date. The refund restriction is reviewed again in November, after the content owner rewrites the partial refund article and QA re-scores the topic.
A decision rule for expanding, holding or restricting
Write the decision rule down before the first review: thresholds for expanding, holding and restricting each topic. Then the meeting applies the rule instead of arguing about it.
A starting rule, to adjust to your own risk appetite:
| Decision | When | What happens |
|---|---|---|
| Expand | Grounding and policy pass rate at or above 98% for two months in a row, and no critical failure | The bot takes a new intent or a wider version of the topic |
| Hold | Pass rate between 96% and 98%, or one good month after a weak one | No scope change; fix the top causes |
| Restrict | Pass rate below 96%, or any critical failure on a money topic | Hand the topic, or the risky part of it, to humans until a re-score clears it |
The thresholds above are our suggestion, not an industry standard. Set them per topic if you need to: the same pass rate may be fine on product questions and too low on refunds. What matters is that the rule exists before anyone sees the numbers.
The 60-minute agenda
Sixty minutes is enough if the report goes out two working days before the meeting and the time goes to decisions, not to reading numbers aloud.
| Minutes | Item | Led by |
|---|---|---|
| 0 to 5 | Headline against goal, and last month’s decisions: done or not | QA lead |
| 5 to 20 | Topics at risk and critical failures, with one transcript each | QA lead |
| 20 to 35 | Top three causes and proposed fixes | Content owner and bot owner |
| 35 to 45 | Last month’s fixes: did the scores move? | QA lead |
| 45 to 55 | Decisions per topic, using the rule | CX leader |
| 55 to 60 | Owners, dates, and criteria changes for next month | QA lead |
Showing a real transcript for each finding matters more than any chart. It turns “grounding fell on refunds” into a sentence the whole room can read, and it settles disputes about whether the scorer was right. Our guide on QA score traceability explains why each score needs the lines behind it.
Weekly, monthly, quarterly: what goes where
The monthly review sits between a weekly operational check and a quarterly report to leadership. Each rhythm has its own job, and mixing them makes all three worse.
| Rhythm | Who | What it covers | What it decides |
|---|---|---|---|
| Weekly (15 minutes) | QA lead, bot owner | Critical failures, new patterns, whether last week’s fixes held | Urgent fixes and handovers |
| Monthly (60 minutes) | The five roles above | The one-page report | Expand, hold or restrict per topic |
| Quarterly | CX leader to leadership | Trend of the headline, scope growth, cost of failures avoided | Budget and roadmap for the AI agent |
For the quarterly report, keep to three numbers leadership can follow over time: the resolved and correct rate, the bot’s share of conversations, and critical failures. Our guide to customer service reporting for executives covers how to present a quality number to a VP or CFO. For the weekly rhythm, our post on QA monitoring cadence has the details.
How to run your first AI agent performance review
Run your first AI agent performance review with one month of scored conversations, the one-page template and a decision rule agreed in advance. Five steps get you there.
Step 1: Score a full month
Score every AI agent conversation for a month against a scorecard that includes the bot’s criteria: grounding, policy limits, scope and escalation. Kaizo’s AutoPilot scores each conversation against your criteria when it resolves.
Step 2: Agree on goals and the decision rule
Set a goal for the headline and each bot criterion, and agree on the thresholds for expanding, holding and restricting before anyone sees the first report.
Step 3: Name the five roles
Put names next to QA lead, bot owner, content owner, support ops lead and CX leader. Book a recurring 60-minute slot in the first week of each month.
Step 4: Send the report two days ahead
Fill the one-page template and send it two working days before the meeting, so the hour goes to decisions.
Step 5: End with owners and dates
Every decision leaves the room with an owner and a date, and the next report opens with whether it happened.
See it in a live demo. Kaizo scores every AI agent conversation on the same scorecard as your team and shows the trends by week, topic and team, ready for your monthly review. Let’s talk.
Frequently asked questions
How often should you review an AI agent’s performance?
Review it monthly for decisions, with a short weekly check for critical failures and fixes. Monthly gives you enough conversations per topic to see a real trend, and it is frequent enough that a problem on a money topic does not run for a quarter. Report the trend to leadership quarterly.
What should an AI agent performance review include?
A headline quality number against its goal, pass rates on the bot’s own criteria, the topics below their goals, any critical failures, the top causes behind failures with a transcript each, whether last month’s fixes worked, and the decisions needed. Keep it to one page and move anything that does not lead to a decision into an appendix.
Can you compare the AI agent with your human agents in the review?
Yes, on the criteria they share, such as a correct answer and the right tone, scored the same way. Compare by topic rather than overall, because the bot usually handles the simpler topics. Do not compare on criteria only one of them has, and never let a bot error count against a human agent’s score.
What happens when an AI agent fails its performance review?
It loses scope until it earns it back. The topics below the threshold, or with a critical failure, go to human agents or get a forced handover. The content owner fixes the source, QA re-scores the topic, and the next review decides whether the bot takes the topic back.
Should the AI agent vendor present the results?
No. The vendor can explain how the bot works and help with fixes, but the results should come from your own QA, on your own scorecard. A vendor presenting its own product’s grade has an interest in a good result, and your leadership will want a number your team produced and can defend.
