AI agent KPIs for customer service should measure whether the AI agent solved the problem correctly, not whether a human stayed out of it. The headline set: correct resolution rate, audited accuracy, policy adherence, handoff quality, repeat contact within 7 days, unsafe auto-action rate and cost per correct resolution. Containment and deflection stay on the dashboard as volume inputs.
AI service quality metrics measure whether an AI agent actually solved the customer’s problem: resolution quality, factual accuracy, escalation quality, repeat contacts, silent abandons, policy adherence and cost per resolved contact. Containment rate and post-AI CSAT, the two AI customer service metrics most teams report, are also the most misleading. Score every conversation instead, with every score linked to the evidence, which is what AI agent QA does.
In short
- Containment counts a frustrated abandon as a win.
- Post-AI CSAT never hears from the customers who quit.
- Repeat contacts and silent abandons catch what containment hides.
- Every score should trace back to evidence in the transcript.
- Correct resolution rate should be the headline AI agent KPI, with cost per correct resolution as its finance twin.
- Vendors define “resolved” differently, so rewrite every vendor number into your own definition before you compare.
Why the two default AI metrics mislead
Every support leader deploying AI is being asked the same question by their board: prove it works. The two numbers that get reached for first are containment rate and post-AI CSAT, because both are easy to pull and both go up and to the right if you squint. They are also the two numbers least likely to describe what actually happened to your customers.
The problem is structural, not a matter of tuning. Containment measures a routing outcome: did a human touch this conversation. Post-AI CSAT measures an opinion, but only the opinions of people who chose to give one. Neither of them looks at the substance of the conversation, which is the only place the answer lives. That is a very different problem from the one solved by the standard method for measuring AI agent performance, and a very different problem from choosing which of the general customer service metrics belong on your dashboard. This is about which numbers are quietly lying to you.
Here is the short version of the argument, metric by metric, before we go into each in detail.
| Metric | What it claims to show | How it misleads | What to measure instead |
|---|---|---|---|
| Containment / deflection rate | The AI handled the conversation successfully | Counts abandonment and stonewalling as success, because no human got involved | Resolution quality plus silent abandon rate |
| Post-AI CSAT | Customers are happy with the AI | Answered by a self-selected minority, and the angriest customers just leave | Repeat contact rate and resolution quality on 100% of conversations |
| Average handle time on AI conversations | The AI is fast | A fast wrong answer is faster than a slow correct one | Factual accuracy and cost per genuinely resolved contact |
| Escalation rate | Fewer escalations means the AI is coping | Suppressing escalations is easy and harmful; the timing and handover matter more | Escalation quality: right moment, full context |
| Volume handled by AI | Scale and ROI | Says nothing about what happened inside that volume | Policy adherence scored against your rubric |
Containment rate: the metric that rewards stonewalling
Containment rate, sometimes called deflection rate, is the share of conversations that ended without a human agent getting involved. It became the default AI metric because it maps directly onto the business case: every contained conversation is a contact you did not pay a human to handle.
The flaw is in the definition. Containment does not ask whether the customer got what they came for. It asks whether anyone else got pulled in. That means several very different outcomes all get counted identically as a win:
- The genuine resolution: the customer asked, the AI answered correctly, the customer left satisfied. This is the outcome the metric was designed to capture.
- The abandonment: the customer asked three times, got a variation of the same unhelpful answer, and closed the window. No human was involved, so it counts as contained.
- The stonewall: the AI could not or would not hand over, kept offering help center articles, and the customer gave up and went to social media or the app store review section instead.
- The displaced contact: the customer closed the chat and called the phone line, or emailed, or opened a new ticket the next day. Contained in one channel, still costing you in another.
Three of those four are failures, and the stonewall is actively worse than doing nothing. Yet a containment number cannot distinguish between them, because the distinguishing information is inside the conversation and containment never looks there.
This is not an argument for abandoning containment. It is a useful capacity and cost input. It is an argument against reporting it as a quality metric, because optimizing containment directly means making it harder for customers to reach a human, which is exactly the behavior nobody intends and everybody recognizes as a customer.
Deflection has the same flaw, plus the confident wrong answer
Deflection rate, the share of contacts that never reach a human queue, has the same blind spot, and both miss the wrong answer the customer believes. A customer told the wrong refund window says thanks and leaves: contained, deflected and, in most vendor reports, resolved. See how to measure containment honestly.
Post-AI CSAT: the metric that never hears from the people who left
CSAT is a good metric in its home context: a human conversation, a reasonable response rate, a stable population answering it. Bolted onto an AI conversation it develops a specific and serious blind spot.
Self-selection at the exact wrong moment
Survey responses come from a minority of customers, and that minority is never random. People who felt strongly enough to respond skew toward the two ends. But the AI failure mode that matters most, the customer who quietly concludes this is not going anywhere and disengages, is precisely the customer least likely to fill in a survey afterward. They did not stay to be surveyed. They are the silent quitters, and post-AI CSAT is structurally deaf to them.
The survey asks about the wrong thing
Even among people who do respond, a satisfaction score conflates whether the answer was correct, whether the tone was pleasant, and whether the outcome was one the customer wanted to hear. An AI that is charming and confidently wrong can score well. An AI that correctly delivers a policy answer the customer dislikes can score badly. Neither result tells you whether the system is working.
The comparison is unfair in both directions
Teams often compare AI CSAT against human agent CSAT and draw conclusions. Those two populations are not comparable: the AI usually takes the simple, high-volume, easily satisfied contacts first, and escalates the hard ones. A flattering AI CSAT can simply mean the AI was given the easy queue. A poor one can mean it was handed the queue nobody could win.
Keep CSAT. Just stop treating it as evidence about your AI. Treat it as one signal among several, weighted by the knowledge that it only hears from the people who stayed.
Which AI agent KPIs should replace containment as the headline?
Seven AI agent KPIs should lead the report: correct resolution rate, audited accuracy, policy adherence, handoff quality, repeat contact within 7 days, unsafe auto-action rate and cost per correct resolution. Each reads the conversation or the systems behind it, so none can be raised by making the human harder to reach.
Containment, deflection, CSAT and handle time stay one level down as capacity and sentiment inputs. The sections below explain each KPI, and the definition table below gives the formulas. For the headline metric in depth, see correct resolution rate.
How do vendors define “resolved” differently?
Vendors use the same word for different tests. Intercom counts a customer who leaves without asking for more help, Zendesk adds a language model review for its verified tier, and Ada requires a relevant, accurate and safe answer. None of these definitions, as documented, checks backend records or contact on other channels.
| Vendor term | Counts as resolved when | Checks the answer was correct? | What it cannot see |
|---|---|---|---|
| Intercom Fin, assumed resolution | The customer “exits the conversation without requesting further assistance” after Fin’s last answer; deducted if the customer reopens the same conversation (Intercom) | No | A wrong answer the customer believed; a return on another channel |
| Intercom Fin, confirmed resolution | The customer confirms the answer helped (“That helped”) | Only the customer’s view | Whether the customer’s view was right |
| Zendesk, contained resolution | The AI agent handled the interaction “without the customer requesting further assistance” (Zendesk) | No; it also covers conversations that failed the verification check | Correctness, backend records |
| Zendesk, verified resolution | As contained, plus a language model reviews the conversation text and confirms the request was resolved | From the transcript text | Whether the action happened in your systems; other channels |
| Ada, automated resolution | No human involved, and the conversation is judged relevant, accurate and safe | Yes, by review of the conversation | Repeat contact after the conversation ends |
So two vendor resolution rates are not comparable until you rewrite both into one definition you own; Fin’s own KPI guide says the definition “varies enormously across vendors” (Fin). Intercom also bills each Fin resolution. And the window is short: Zendesk ends a messaging conversation 2 hours after the first message by default and an email conversation 72 hours after the first email, so a failure that surfaces on day 4 never touches the vendor’s number. Our AI agent resolution audit shows how to report a corrected rate next to the vendor’s.
Resolution quality: the anchor metric
If you replace containment with one thing, replace it with resolution quality: did this conversation actually solve the customer’s problem. It is the anchor of the whole stack because every other metric here is either an input to it or a check on it.
What it is. A judgment, made against the transcript, about whether the customer’s underlying issue was addressed. Not whether a ticket was closed, not whether a response was sent, not whether the intent was matched. Whether the thing the customer needed to happen, happened.
How to measure it. As a scored criterion on your QA rubric, applied to AI-handled conversations exactly as it is applied to human-handled ones. The criterion needs to be written so that a reviewer and an automated evaluator would read it the same way: state what evidence in the transcript counts as resolution for each major contact type. Refund requested and refund confirmed. Address changed and change verified back to the customer. Question asked and correct answer given with the customer acknowledging it.
What a bad number looks like. The signal to watch is not an absolute threshold, it is the gap between resolution quality and containment. If a large share of conversations are contained but a much smaller share are scored as resolved, that gap is the size of your problem, expressed in conversations. It is also the number worth putting in front of a board, because it is the honest version of the one they were shown.
Note that this is deliberately stricter than first contact resolution, which infers resolution from the absence of a follow up. FCR is a useful proxy. Resolution quality reads the transcript instead of inferring from silence, and silence is exactly what an abandoning customer produces.
Factual accuracy and hallucination rate
An AI agent that invents a refund window, misstates a delivery timeline, or confidently describes a feature you do not have is not a quality problem, it is a liability problem. Factual accuracy is the metric that catches it, and it is the one most teams have no measurement for at all.
What it is. The share of AI responses containing a factual claim that is checkable against your knowledge base, policy documents, or order data, and the share of those claims that are correct. Hallucination rate is the inverse: claims that are confidently stated and unsupported by any source.
How to measure it. Every factual assertion in a conversation gets checked against the grounded source of truth. This is only practical automatically, at volume, which is one of the strongest arguments for scoring every conversation rather than sampling: a hallucination in a conversation nobody read costs exactly as much as one in a conversation somebody did. Track it separately by contact type, because accuracy is rarely uniform. The billing questions and the shipping questions usually behave very differently.
What a bad number looks like. Any nonzero rate on claims that carry legal, financial, or safety weight is a bad number, regardless of how small it is. On lower-stakes claims, the direction is what matters: a hallucination rate that climbs after a knowledge base change or a model update is telling you something specific and urgent about that change.
One caution: an AI agent’s confidence in its own answer is not evidence that the answer was true. Accuracy has to be checked against your knowledge base and policies, with the evidence for each verdict shown, which is the principle behind a well-run LLM as a judge evaluation.
Escalation quality: not how often, but how well
Escalation rate is reported as though lower is better. It is not. Escalation is a feature, and an AI that never escalates is not a strong AI, it is a trapped customer. What matters is escalation quality: whether the handover happened at the right moment and arrived with the right context.
Timing
The two failure modes sit on either side of correct. Escalating too early wastes the automation entirely and irritates a customer who had a simple question. Escalating too late, after three or four failed loops, means the human inherits a customer who is already angry and a conversation that has to be restarted. Score the moment: was there an identifiable turn where a competent handover should have happened, and did it happen there?
Context transfer
The most expensive escalation failure is the one where the human agent opens the conversation and has to ask the customer to explain everything again. That single behavior undoes the goodwill of the entire AI interaction. Score whether the handover carried the customer’s stated issue, what was already attempted, and any account or order context the AI had already retrieved.
Correctness
Was escalation the right call at all? An AI that escalates every ambiguous message is generating human workload for nothing, which shows up in cost per resolved contact rather than in quality scores. Both directions need scoring, or you will optimize one into the other.
As a single KPI, score every handoff pass or fail on right moment, full context and right call; handoff quality is the share that pass all three. See scoring the AI-to-human handoff.
What a bad number looks like. A falling escalation rate alongside a falling resolution quality score is the clearest bad signal in this entire article. It means the AI is holding onto conversations it cannot finish.
Repeat contact rate and silent abandon rate
These two are grouped together because they are your cheapest lie detectors. Both catch failures that containment scores as successes, and neither requires a customer to fill anything in.
Repeat contact rate after an AI conversation
What it is. The share of customers who come back within a defined window, in any channel, about the same underlying issue after an AI-handled conversation.
How to measure it. Link contacts by customer and by issue, not by ticket, and specifically look across channels. A customer who abandoned a chat and then phoned is the single most important case to catch, and a per-channel view will miss them completely. Set the window to match your product: same day for a delivery issue, a week or more for a billing dispute. For the headline KPI, fix one 7-day window so the number compares week to week; topic windows stay useful for diagnosis.
What a bad number looks like. A repeat contact rate that rises after AI handling compared with the equivalent human-handled contacts is a red flag, whatever its absolute value. It means containment moved work rather than removing it.
Silent abandon rate
What it is. The share of AI conversations the customer exits without resolution and without escalating. No survey, no complaint, no follow up. These conversations look identical to successes in every routing-based metric.
How to measure it. Identify conversations that ended on a customer turn, or ended with the AI’s last message unanswered, and score whether the issue had been resolved at that point. Splitting satisfied departures from silent quits is a transcript judgment, which is exactly why this metric does not exist on most dashboards: it cannot be counted, it has to be read.
What a bad number looks like. Any silent abandon rate that is a meaningful fraction of your containment rate means your containment number is inflated by exactly that much. Reporting containment without it is reporting a number you know is wrong.
Unsafe auto-action rate: the KPI for AI agents that take actions
Unsafe auto-action rate is the share of actions an AI agent carried out on its own, such as refunds, cancellations, credits or account changes, that broke a rule: over a limit, a required check skipped, the wrong record, an action nobody asked for, or an action claimed but never done in the backend. Accuracy scores the text; this KPI scores what the agent did.
How to measure it. Reconcile every executed action, not a sample, with the policy limits and the conversation that triggered it. Track override rate (actions a human blocked) next to it: a falling unsafe rate with a rising override rate means humans are catching the problem, not that it went away. Any unsafe action on payments, account access or personal data is an incident with a named owner. The rules belong in your AI agent guardrails.
Policy adherence and cost per genuinely resolved contact
The last two close the loop between quality and the business case.
Policy adherence
What it is. Whether the AI followed the rules it is required to follow: verifying identity before disclosing account details, giving mandatory disclosures, refusing out-of-policy discounts, handling a vulnerable customer or a complaint disclosure correctly, staying inside regulatory language where it applies.
How to measure it. These are binary, evidence-checkable criteria, which makes them the easiest thing on this list to score automatically and the most damaging thing to sample. Run them as pass or fail checks on every conversation, and treat each fail as an incident to review rather than a percentage to average away. This is the same rubric logic your human team is already scored against, which is the point: your internal quality score should cover the AI too, or you have two standards and no comparison.
What a bad number looks like. On identity verification and regulated disclosures, one fail is a bad number. Do not average these.
Cost per genuinely resolved contact
What it is. Total cost of the AI conversations divided by the number of conversations that were actually resolved, not the number contained.
How to measure it. Take your resolution quality score, apply it to your contained volume, and use the resulting count as the denominator. Include the downstream cost the AI created: repeat contacts, the extra handle time on escalations that arrived without context, and the human time spent cleaning up.
What a bad number looks like. If cost per genuinely resolved contact is not meaningfully better than your human-handled equivalent, the deployment is not paying for itself no matter how good the containment chart looks. This is the number that turns a quality argument into a finance argument, which is usually the argument that gets budget.
The AI agent KPI definition table (copy this)
Each headline KPI needs a formula and a data source, or two teams will compute it two ways. The targets are starting points for a first quarter, not benchmarks: replace them with your own baseline after four weeks of scoring.
| KPI | Formula | Data source | Example target (starting point) |
|---|---|---|---|
| Correct resolution rate | AI conversations with a right answer, the action done, policy held and no repeat contact within 7 days, divided by all AI-handled conversations | QA scoring of transcripts, backend records, contact history | Within 10 points of the vendor-reported resolution rate, then raise it each quarter |
| Audited accuracy | Factual claims confirmed correct, divided by factual claims checked | Scoring against knowledge base, policy documents and order data | 98% overall; 100% on money, legal and safety claims |
| Policy adherence | Conversations passing every mandatory check that applied, divided by conversations where a check applied | Pass or fail scorecard criteria (identity, disclosures, refund limits) | 100% on identity and regulated disclosures; every fail reviewed |
| Handoff quality | Handoffs passing timing, context and correctness, divided by all handoffs | Scorecard escalation criteria plus helpdesk handoff events | 90% of handoffs pass; under 5% of customers asked to repeat themselves |
| Repeat contact within 7 days | Customers back about the same issue on any channel within 7 days, divided by AI conversations marked resolved | Contact history matched by customer ID across channels | At or below the human-handled rate for the same contact type |
| Unsafe auto-action rate | AI-executed actions that broke a rule, divided by all AI-executed actions | Action logs reconciled with policy limits and the triggering conversation | Zero on payments and account access; under 0.1% elsewhere, each case reviewed |
| Cost per correct resolution | AI cost plus the cost of the human follow-ups it caused, divided by correct resolutions | Platform invoices, workforce cost, contact history | Below your cost per correct human resolution for the same contact type |
Cut every row by contact type, because an overall average hides the one intent that is broken. Use the same definitions for your human team, so AI and human rows can sit on one dashboard; where a KPI has no human twin, such as unsafe auto-action rate, show the nearest control (out-of-policy refunds and credits by agents).
Worked example: one month, two headline numbers
An illustration with round numbers, not a benchmark. An AI agent handles 10,000 conversations in a month. The vendor dashboard shows 70% containment (7,000 conversations) and 65% resolution (6,500).
The team audits 400 of the “resolved” conversations. 14% gave a wrong answer or claimed an action that never happened, and another 8% came back about the same issue within 7 days. That leaves 78% correct.
| Number | Calculation | Result |
|---|---|---|
| Correct resolutions | 6,500 x 78% | 5,070 |
| Correct resolution rate | 5,070 / 10,000 | 50.7% (vendor says 65%) |
| Follow-ups the AI caused | 6,500 x 22% | 1,430 |
| Cost per vendor resolution | $13,000 AI cost / 6,500 | $2.00 |
| Cost per correct resolution | ($13,000 + 1,430 x $6 human follow-up) / 5,070 | $4.26 |
Same month, same conversations. One report says the AI resolves 65% of contacts at $2.00; the other says it resolves half of them at more than twice the cost. With a sample of 400, the 78% carries a margin of about 4 points either way (95% confidence); scoring every conversation removes that margin.
Which dashboards show containment rate, CSAT and handle time
Most helpdesks and AI agent platforms already report containment rate, CSAT and average handle time in their built-in analytics. What those dashboards rarely show is whether each conversation was actually resolved, correct and compliant.
| Metric | Where it usually comes from | What it cannot tell you |
|---|---|---|
| Containment or deflection rate | The AI agent platform or helpdesk analytics | Whether the customer was helped or gave up |
| CSAT | Post-conversation surveys in the helpdesk | What the customers who did not answer experienced |
| Average handle time | Helpdesk or contact center reporting | Whether a fast answer was a right answer |
| Resolution quality, accuracy, policy adherence | A QA platform that scores the transcripts | (This is the layer the other three are missing) |
So the practical setup is two layers. Keep the operational numbers from the tools that produce them, and add a quality layer that reads every conversation, for human and AI agents alike. Kaizo is built for that second layer: it scores AI agent conversations against the same scorecard as your team’s, so you can see quality by team and by week, human and AI side by side. For the containment side in more depth, see what containment rate is and why it can mislead.
Why AI service quality metrics need every conversation
Read back through the metrics above and one thing is true of every single one: it requires reading the conversation. Resolution quality is a judgment about the transcript. Hallucination rate is a check of claims against sources. Escalation quality is an assessment of a specific turn. Silent abandon is the absence of a signal, which means it can only be found by looking at conversations where nothing happened.
That has two consequences most measurement programs have not absorbed yet.
Sampling does not work here
Traditional QA reviews a small percentage of conversations, and for a human team of a known size that is a defensible statistical compromise. It is not defensible for AI, for two reasons. First, AI failures are systematic rather than distributed: a bad knowledge article or a policy edge case produces the same failure every time it is hit, so a sample either catches the whole cluster or misses it entirely. Second, AI handles far more volume, so the same small sample is a smaller window on a bigger operation. Scoring every conversation is the precondition rather than the point: it removes the selection bias that makes the other numbers meaningless. It is not a matter of watching anyone, it is a matter of not letting a sampling rule decide what your metrics say.
You should not have to take a score on trust
Insist that every score links back to the exact evidence in the transcript that produced it, that any score can be challenged and adjudicated by a human, and that the grader can be run against a set of conversations your own reviewers scored so you can see the agreement rate criterion by criterion before you report anything upward. A score you can trace is a score you can challenge. A score you cannot trace is a marketing claim with a number attached.
Kaizo scores AI agents and your human team against one rubric you define, with every score traced back to the transcript and testable against your own reference set. It applies that rubric across all your conversations rather than a chosen slice. Practically, it means QA for AI agents and QA for humans stop being two separate programs producing two incomparable sets of numbers.
How to measure AI customer service performance
Measure AI customer service performance in five steps: define resolution per contact type, score every AI conversation against one scorecard, add the silent signals, compare with your human team, and report the gap.
- Define resolution per contact type. Write down what evidence in the transcript counts as resolved for your top contact reasons: refund confirmed, address change verified, correct answer acknowledged.
- Score every AI conversation against one scorecard. Use the same criteria as for your human team, plus the bot-specific ones: factual accuracy, policy fidelity, staying in scope and escalating at the right moment.
- Add the silent signals. Track repeat contacts across channels and silent abandons, because neither shows up in containment or CSAT.
- Compare with your human team. Put AI and human quality on the same chart, by contact type, so you are not comparing the easy queue with the hard one.
- Report the gap. Show leadership containment next to resolution quality, and cost per genuinely resolved contact next to cost per contained contact.
Test the scoring itself before you report it: run it on conversations your reviewers already scored and check the agreement per criterion. Our guide to measuring AI agent performance covers the method in more detail.
When is this KPI set not for you?
If your AI agent handles a few hundred conversations a month, answers only FAQs and takes no actions, seven KPIs are overhead: read a weekly sample and track repeat contacts.
Without a written scorecard, write one first (AI agent scorecard). An AI that only drafts replies a person sends is agent assist, so the human-team KPIs apply. To start, score 100 random “resolved” conversations from last month for right answer, action done, policy held and repeat contact within 7 days, and report the result next to the vendor’s rate (margin about 10 points). For alert thresholds, see AI agent monitoring.
Measure what your AI agents actually did
See how resolution quality sits next to containment in a live demo on realistic sample data, graded against a rubric like yours. Every score traces to the evidence in the transcript, and in your own rollout you check it against conversations your reviewers have already scored.
Let's talk See AI agent QA See Kaizo Insights
Frequently asked questions
What are the most important AI customer service metrics?
Correct resolution rate, audited accuracy, policy adherence, handoff quality, repeat contact within 7 days, unsafe auto-action rate and cost per correct resolution, with silent abandon rate as a diagnostic. These describe what actually happened in the conversation and in your systems. Containment rate and post-AI CSAT, the two most commonly reported, describe routing and self-selected opinion instead, which is why they mislead.
What is the difference between resolution rate, deflection and containment?
Containment is the share of conversations that ended without a human. Deflection is the share of contacts that never reached a human queue. Resolution rate is the vendor’s own test, from “the customer left without asking for more help” to an automated transcript review. None checks backend records or other channels, which is why correct resolution rate sits above them.
What is containment rate and why is it misleading?
Containment rate, also called deflection rate, is the share of conversations an AI handled without a human getting involved. It is misleading because it measures whether anyone else got pulled in, not whether the customer got helped. A customer who asked three times, got nowhere, and closed the window in frustration counts as contained, and so does a customer who gave up and phoned instead.
How should you measure CSAT for AI agents and chatbots?
Treat CSAT as one signal among several, not as the verdict. Post-AI CSAT is answered by a small self-selected group, and the customers your AI failed hardest are the ones least likely to stay and answer a survey. It also conflates correctness with tone, so a confidently wrong AI can score well. Compare it against transcript-based measures rather than treating it as evidence on its own.
How do you measure whether an AI agent actually resolved an issue?
Score resolution as a criterion on your QA rubric, applied to the transcript rather than inferred from ticket status. Define, per contact type, what evidence in the conversation counts as resolution: the refund confirmed, the change verified back to the customer, the correct answer acknowledged. Then compare that resolved share against your containment rate. The gap between the two is the size of the problem.
What is silent abandon rate?
It is the share of AI conversations a customer exits without resolution and without escalating: no complaint, no survey, no follow up. These conversations are invisible to every routing-based metric because no human was involved, so they inflate containment. Finding them means reading conversations that ended on a customer turn and judging whether the issue had actually been addressed.
What resolution rate can AI customer support achieve?
There is no reliable industry benchmark, because vendors report different things under the same name: many count a contained conversation as resolved. The rate that matters is the share of AI conversations that a transcript review scores as genuinely resolved for your contact types. Measure that first, then compare it with containment; the gap tells you how far the headline number overstates the real one.






