An AI agent resolution rate counts conversations the vendor marks as resolved: no handoff, no follow-up within a time window. It does not check that the answer was right, the action was done, or the customer stayed away. To audit it, sample “resolved” conversations, judge each against written criteria and backend records, and report a corrected rate next to the vendor’s.
In short
- A vendor “resolution” usually means the customer did not ask for more help before a timer ran out; a confident wrong answer and a customer who gave up both pass that test.
- Four failures hide inside resolved conversations: a wrong answer the customer accepted, a customer who gave up, a customer who came back on another channel, and an action the agent claimed but never completed.
- A correct resolution needs four yeses: right answer, allowed by policy, action done in the system of record, and no repeat contact about the same issue within 7 days.
- A random sample of about 400 resolved conversations a month gives a corrected rate within roughly 5 points, which is enough to report next to the vendor’s number.
- Report both numbers, the failure mix and the cost per correct resolution, and send each failure type to an owner.
This guide is a spoke of our AI agent evaluation hub, written for the Head of CX or AI lead who signs off on the AI agent’s results. If you need the basics of the metric that usually sits next to resolution rate first, start with containment rate.
What does “resolved” mean in an AI agent resolution rate?
In most vendor reports, “resolved” means the AI agent answered and the customer did not ask for more help, or ask for a human, before the conversation timed out. Some vendors add a model check on the transcript. None of these definitions, as documented, looks at backend records or at the customer contacting you again on another channel.
The definitions are public, and they are worth reading word for word before you trust a dashboard:
| Vendor definition | When it counts as resolved | What it cannot see |
|---|---|---|
| Intercom Fin, “assumed resolution” | The customer “exits the conversation without requesting further assistance” after Fin’s last answer; deducted if the customer returns to the same conversation (Intercom) | Whether the answer was correct; returns on another channel or in a new conversation |
| Zendesk, “Contained resolution” | The AI agent handled the interaction “without the customer requesting further assistance” (Zendesk) | Correctness; this tier also holds conversations that failed the verification check |
| Zendesk, “Verified resolution” | As contained, plus a large language model reviews the conversation text and confirms the request was resolved | Backend records and other channels, since the check reads the conversation text |
| Ada, “automated resolution” | Resolved without a human and judged relevant, accurate and safe by an LLM review (Ada) | Repeat contact after the conversation; the LLM review runs on a sample of finished conversations |
Two things follow. First, “resolved” is a billing and reporting definition set by the vendor, often tied to what you pay. Second, the definition moves: Intercom now reports “outcomes” next to resolutions and writes that “success” stopped being binary (Intercom blog, March 2026), and Zendesk introduced resolution tiers on May 18, 2026 (Zendesk legacy definition). An audit gives you a definition you own.
What is the difference between resolved and correctly resolved?
A resolved conversation ended without a handoff or follow-up. A correctly resolved conversation also gave the right answer, completed the right action, stayed inside policy, and kept the customer from coming back about the same issue. The gap between the two is your real automation rate, and it is usually invisible on the vendor dashboard.
The vendor asks: did the conversation end quietly? Your audit asks: did the problem go away? The second question needs evidence from outside the transcript. Our post on AI agent silent failures covers the wider family of failures that throw no error, and why AI conversations fail covers the causes behind them.
Which failures hide inside resolved conversations?
Four failure types pass as resolutions: a wrong answer the customer accepted, a customer who gave up, a customer who came back on another channel, and an action the agent claimed but did not complete. Each one looks like a quiet, successful conversation in the transcript, and each needs a different check to find.
| Failure type | What happened | Why it counts as resolved | How the audit catches it |
|---|---|---|---|
| Wrong answer accepted | The agent gave a confident, incorrect policy, price or step; the customer believed it and left | The customer did not ask again | Compare the answer with the policy or knowledge article valid on that date |
| Customer gave up | The agent looped, misread the question or hid the route to a human; the customer closed the window | Leaving without asking for help is the definition of an assumed resolution | Read whether the actual question was answered; look for frustration, repeated rephrasing, an abrupt end |
| Came back on another channel | The customer emailed, phoned or opened a new chat about the same issue | Vendors deduct returns to the same conversation at most, not to other channels | Match the customer ID across channels for 7 days |
| Action claimed but not done | The agent said “your refund is processed” or “your address is updated”, but the system shows nothing, or something else | The transcript reads like a perfect resolution | Check the refund, order, billing or CRM record for the change, amount and date |
The last type is the most expensive to miss, and no transcript-only review, human or LLM, can catch it. AI agent guardrails on actions reduce how often it happens; the audit tells you whether they work.
How do you design a resolution audit sample?
Draw a random sample of about 400 conversations the vendor marked as resolved in one month, stratified by intent, with extra conversations from high-risk intents such as refunds, cancellations and account changes. Hide the vendor status from reviewers, and double-score 10% to measure agreement.
- Fix the period and the population. One calendar month, every conversation with vendor status “resolved”.
- Pick at random inside each intent, in proportion to volume. Hand-picked “interesting” conversations bias the rate downward.
- Oversample high-risk intents. Add at least 50 conversations from intents where the agent takes an action or touches money; weight them back to their real share for the overall rate.
- Size for the decision. 400 conversations give a margin of about ±5 points at 95% confidence (see the confidence intervals for a proportion in the NIST engineering statistics handbook). 100 per intent gives about ±10 points: enough to spot a broken intent. The math is in our guide to QA sampling statistics.
- Blind the reviewer. Show the transcript and the backend data, not the vendor label.
- Measure agreement. A second reviewer scores 40 of the 400. If verdicts differ more than 1 time in 10, fix the criteria before you report.
- Wait for the window. Pull the sample at least 7 days after the month ends.
What are the criteria for a “correct” resolution?
A conversation is a correct resolution only when four checks all pass: the answer was right and complete, it stayed within policy, any action the agent promised is visible in the system of record, and the customer did not contact you again about the same issue within 7 days. One failed check makes the verdict “not correct”.
Write the criteria down before the first review, with an example of a pass and a fail for each:
| Check | Pass when | Evidence | Fail example |
|---|---|---|---|
| Right answer | Matches the policy, price or procedure valid on the conversation date, and answers every part of the question | Knowledge base version, policy document, price list | Quotes a 30-day return window when the policy says 14 |
| Allowed | The agent stayed within what it may say and do (no exceptions it cannot grant, no advice outside scope) | Written agent scope and policy | Promises a goodwill credit only a team lead may approve |
| Action done | Every action the agent stated is in the system of record, with the right amount, item and date; “no action needed” passes | Order, billing, refund or CRM record | “Refund issued” but no refund record exists |
| No repeat contact | No contact from the same customer about the same issue, on any channel, within 7 days | Contact history joined on customer ID | Customer emails 2 days later: “the chat said it was cancelled, I was charged again” |
Keep a fifth verdict, “cannot judge”, for conversations where a record is missing or the customer is anonymous. Leave them out of the rate and report how many there were; if they pass 5% of the sample, fix the data before the next audit. The right answer depends on your policies being written and current, which is the subject of AI agent policy management.
How do you check backend records for claimed actions?
List every action the agent stated in the conversation, then look each one up in the system that owns it: the refund in billing, the cancellation in the subscription system, the address in the order record. The agent’s own tool log is not enough, because a tool call can report success while the system of record shows nothing.
A practical sequence for each sampled conversation:
- Extract the claims. Highlight every sentence where the agent says it did something: “I have refunded”, “your plan is cancelled”, “I have updated your address”.
- Join on identifiers. Use the customer ID, order number or email from the conversation to find the record.
- Compare four fields. Did it happen, to the right item, for the right amount, before or soon after the conversation ended?
- Note mismatches precisely. “Refund of $40 promised, $0 in billing” is fixable; “action failed” is not.
No access to the system of record? Ask its owner for a read-only export filtered on the 400 sampled customer IDs.
How do you measure repeat contact after an AI resolution?
Count a repeat contact when the same customer reaches you again about the same issue, on any channel, within 7 days of the AI conversation ending. Match on a customer identifier across chat, email, phone and social, then read the follow-up to confirm it is the same issue, not a new one.
Why 7 days: the vendor windows are short and channel-bound. Zendesk ends a messaging conversation 2 hours after the first message by default and an email conversation after 72 hours; Intercom deducts a resolution only when the customer returns to the same conversation. Many failures surface later: the refund that does not arrive in 3 to 5 business days, the second charge at the next billing run. Seven days catches most of them; for billing intents, extend the window to the next invoice date.
If contact history is split across tools, start with the identifiers you have (account ID, email address) and note in the report that some repeats will be missed. Following one customer across channels is the core of customer journey QA.
The resolution audit worksheet (copy this)
The worksheet below is one row per sampled conversation. Fill the four checks, then let the verdict rule decide; reviewers do not choose the verdict by feel. The rows are an illustration of how each failure type looks once filled in, not customer data.
| # | Conversation | Intent | Vendor status | Answer correct? | Action done? | Repeat contact within 7 days? | Verdict | Failure type | Evidence note |
|---|---|---|---|---|---|---|---|---|---|
| 1 | C-10482 | Order status | Resolved (assumed) | Yes | No action needed | No | Correct | None | Tracking link matches carrier record |
| 2 | C-10517 | Returns | Resolved (assumed) | No | No action needed | No | Not correct | Wrong answer accepted | Said 30-day window; policy v12 says 14 days |
| 3 | C-10533 | Cancel plan | Resolved (confirmed) | Yes | No | Yes, email after 2 days | Not correct | Action claimed but not done | “Cancelled” in chat; subscription still active, charged again |
| 4 | C-10561 | Billing question | Resolved (assumed) | Partly | No action needed | Yes, phone after 1 day | Not correct | Came back on another channel | Answered tax line, not the duplicate charge asked about |
| 5 | C-10590 | Account access | Resolved (assumed) | No answer given | No action needed | No | Not correct | Customer gave up | Customer rephrased 3 times, then left |
| 6 | C-10612 | Refund | Resolved (assumed) | Yes | Record missing | No | Cannot judge | None | Billing export lacks this order; request data |
Verdict rule. Correct = answer correct (yes), action done (yes or no action needed), no repeat contact, and allowed by policy. Any “no” = not correct. Any missing evidence = cannot judge. When two failures apply, record the one the customer felt first.
Add an “Allowed?” column if your agent can grant exceptions, and keep each evidence note specific enough to check in two minutes.
Worked example: vendor-reported vs audited resolution rate
In this illustration, a vendor reports a 68% resolution rate on 10,000 conversations. An audit of 400 resolved conversations finds 69% of the judgeable ones correctly resolved, so the corrected resolution rate is about 47%, with a 95% range of roughly 44% to 50%.
Illustration (not customer data). An AI agent handles 10,000 conversations in October. The vendor marks 6,800 as resolved: a 68% resolution rate. The team audits a random sample of 400 resolved conversations with the worksheet above.
| Audit verdict | Conversations | Share of sample |
|---|---|---|
| Correct resolution | 268 | 67% |
| Wrong answer accepted | 36 | 9% |
| Customer gave up | 40 | 10% |
| Came back on another channel within 7 days | 32 | 8% |
| Action claimed but not done | 12 | 3% |
| Cannot judge (left out of the rate) | 12 | 3% |
| Total | 400 | 100% |
The calculation, step by step:
- Judgeable conversations: 400 minus 12 = 388.
- Audited correct share: 268 / 388 = 69.1%.
- Margin of error: 1.96 x √(0.691 x 0.309 / 388) = ±4.6 points, so the correct share lies between about 64.5% and 73.7%.
- Corrected resolved conversations: 6,800 x 69.1% = about 4,700.
- Corrected resolution rate: 4,700 / 10,000 = 47.0%, with a range of about 43.8% to 50.1%.
- Cost per correct resolution: at an illustrative price of $1.00 per billed resolution, 6,800 billed resolutions cost $6,800. Divided by 4,700 correct resolutions, each one costs about $1.45, not $1.00.
Read the failure mix as a to-do list: wrong answers go to the knowledge and prompt owner, give-ups to escalation design, and the 12 claimed actions come first, because each is a customer told something false about their money or account.
How do you report a corrected resolution rate next to the vendor’s number?
Show the vendor’s resolution rate and your audited rate side by side, every month, with the definitions, the sample size, the margin of error and the failure mix underneath. Do not replace the vendor number: the gap between the two is the finding, and its trend tells you whether fixes are working.
A one-screen monthly report:
| Line | October (illustration) |
|---|---|
| Vendor-reported resolution rate | 68% (vendor definition, linked) |
| Audited correct share of resolved conversations | 69% ±4.6 points (n = 388) |
| Corrected resolution rate | 47% (range 44% to 50%) |
| Failure mix | Gave up 10%, wrong answer 9%, other channel 8%, action not done 3% |
| Cost per correct resolution | $1.45 vs $1.00 per billed resolution |
| Worst intent | Cancel plan: 58% correct (n = 60) |
Keep the definitions stable, show the range so a 2-point move inside the noise does not start a debate, and put the corrected rate where leadership already sees the vendor’s number. It also belongs in your wider set of AI agent performance metrics and in the weekly view described in AI agent monitoring.
When is a full resolution audit not for you?
A formal audit is overkill when your AI agent handles a few hundred conversations a month: read them all instead of sampling. It also matters less when the agent only answers informational questions with no actions and little money at stake; there, a lighter accuracy check and a repeat-contact count may be enough.
It is also not a substitute for scoring every conversation. A 400-conversation sample tells you the rate; it will rarely show you a failure that hits 1 in 500 customers. If you need to find those individually, you need full coverage, which our hub on evaluating AI agents in production explains with sample-size math.
How to start a resolution audit this week
You can run the first audit in one week with a spreadsheet, a reviewer and read-only access to two systems. The goal of week one is not a perfect number but a first corrected rate and a list of the failures behind it.
- Monday: write down your vendor’s resolution definition, and your four checks with a pass and a fail example each.
- Tuesday: export last month’s resolved conversations, stratify by intent, and draw 400 at random.
- Wednesday: request read-only exports from billing or the order system, and contact history for the sampled customer IDs.
- Thursday and Friday: fill the worksheet; double-score 40 conversations and compare verdicts.
- Next Monday: compute the corrected rate and the cost per correct resolution, and send each failure type to an owner.
Once the audit shows where resolutions break, the next step is catching those failures on every conversation, not only in a monthly sample. Kaizo’s AutoPilot scores every resolved conversation against your own scorecard, and every score links to the evidence in the conversation; our page on QA for AI agents shows how it works. See it on your own AI agent’s conversations. Book a demo.
Frequently asked questions
Is resolution rate the same as containment rate?
No. Containment rate counts conversations that ended without a human handoff. Resolution rate, as vendors define it, usually adds a condition that the customer did not ask for more help within a time window. Neither checks that the answer was correct, which is why both need an audit next to them.
What is a good AI agent resolution rate?
There is no reliable benchmark, because vendors define resolution differently and change their definitions over time. Compare your own corrected resolution rate month over month instead. A lower vendor rate with a small gap to the audited rate is healthier than a high rate that halves under audit.
Can an LLM check replace a human resolution audit?
It can cover more conversations, and some vendors already use one to verify resolutions. But a check that reads only the conversation text cannot see backend records or later contacts on other channels. Calibrate any LLM judge against human verdicts, and keep the backend and repeat-contact checks in the audit.
How often should you audit AI agent resolutions?
Monthly is a good default, with an extra audit after any change to the agent’s prompts, knowledge, actions or scope. Pull the sample at least 7 days after the month ends so every conversation has its full repeat-contact window. High-risk intents can justify a weekly check.
Should you pay for assumed resolutions that fail your audit?
That depends on your contract, which is why the audit matters at renewal. Your corrected rate and cost per correct resolution give you evidence for the conversation with the vendor. Ask how the vendor’s definition handles returns on other channels and actions that never happened.
Who should own the resolution audit?
The team that answers for customer outcomes, usually CX quality or support operations. The owner keeps the definitions stable and routes each failure type to whoever can fix it: knowledge, engineering or escalation design.
