AI Agent Evaluation for Support: Methods and Metrics

AI agent evaluation is how a support team proves its AI agent gives correct, allowed and helpful answers. It has two stages: evals before launch, where you run test conversations against written success criteria, and production evaluation, where you score real customer conversations against a scorecard. You need both, a calibrated judge, and a named owner.

In short

• Pre-launch evals tell you whether a change is safe to ship; production evaluation tells you what the AI agent actually did to customers.
• Measure correct resolution, policy adherence, action accuracy, escalation quality and tone; treat containment as context, never as the quality score.
• Use an LLM judge for volume and human reviewers for calibration and high-risk cases, and check their agreement criterion by criterion.
• Small samples miss rare but costly failures: at 40,000 conversations a month, a 1% sample has a 45% chance of showing zero cases of a failure that hits 80 customers.
• Give every metric an owner, a threshold and a cadence; the framework table below is a starting point you can copy.

This guide is for the support leader who answers for the agent once it talks to customers, not for the engineers who build it. For the basics, see how to QA AI agents; if you are still deciding how far to automate, start with agentic customer service.

What is AI agent evaluation in customer service?

AI agent evaluation in customer service is the repeatable process of checking an AI agent’s conversations against written success criteria: was the answer correct, was it allowed by policy, did the promised action really happen, did the agent hand over when it should, and was the tone right. It runs before launch on test cases and after launch on real conversations.

Your AI team will use these terms from developer guides:

The last row matters most for support. A transcript where the agent says “your refund is on its way” is not the same as a refund in the system. Practitioners running support agents describe exactly this failure: the agent reports an action as done when the API call behind it failed.

What does Anthropic’s “Demystifying evals for AI agents” mean for support teams?

Anthropic’s engineering post Demystifying evals for AI agents is the guide most people search for by name. For support teams it boils down to five points: start small with real failures, check outcomes and not only words, measure consistency, test both sides of every decision, and keep reading transcripts.

1. Start with 20 to 50 real failures. The post calls 20 to 50 simple tasks drawn from real failures “a great start” and names the support queue as a source. Your QA team already holds them.
2. Grade the outcome, not the narration. The post’s refund example combines a state check (ticket resolved, refund processed), checks on the agent’s actions (identity verified, refund within limits) and an LLM rubric for empathy and clarity.
3. Measure consistency with pass^k, the chance that all k runs of a task succeed. The post notes it “especially matters for customer-facing agents where users expect reliable behavior every time.” An agent that passes a task 75% of the time passes all 3 runs only about 42% of the time.
4. Test both directions. Check when the agent should act and when it should not. A set full of “should refund” cases produces an agent that refunds too often.
5. Read the transcripts. “You won’t know if your graders are working well unless you read the transcripts and grades from many trials.”

The post also calls production monitoring reactive: customers meet the problem before you do. Support teams need both stages.

Pre-launch evals vs production evaluation: what is the difference?

Pre-launch evals run your AI agent on test conversations before a change goes live, to block a bad prompt, model or knowledge update. Production evaluation scores the real conversations customers had, so you can find failures your test set never imagined. Pre-launch is a gate; production is the ongoing quality check.

The two stages feed each other: every failure found in production becomes a new test case. Simulation, where a second model plays the customer, sits on the pre-launch side; benchmarks such as τ-bench use it for retail and airline support tasks.

How do you write evals for an AI agent?

To write evals for an AI agent, collect 20 to 50 real conversations where the agent or your team got it wrong, write down what a correct outcome looks like for each, add cases where the right move is to refuse or escalate, give each criterion its own grader, and run every case several times before each change.

The support team defines it; the AI side wires it up:

1. Pull the cases from QA. Failed or escalated conversations from the last 90 days, grouped by intent.
2. Write the expected outcome in one sentence. “Refuses the refund, quotes the 30-day policy, offers store credit, does not escalate.”
3. Add the mirror cases. For every “should refuse” case, a “should do it” case; for every “should escalate” case, one where escalation would be wrong.
4. Split the criteria. Correctness, policy, action, escalation and tone each get their own check; Anthropic’s post recommends one judge per rubric dimension.
5. Check the system, not the sentence. Where the agent takes an action, the grader reads the order or helpdesk record, not the agent’s claim.
6. Run each case 3 to 5 times and count it as passed only when all runs pass. That is pass^k in practice.
7. Keep two sets. A regression set that should pass close to 100% of the time, and a harder capability set for the intents you plan to automate next.

The minimum version, as practitioners describe it, is a few dozen real tickets that went wrong, rerun every time the prompt changes. Our guide to AI agent testing covers test design, regression runs and simulation in depth.

Which AI agent evaluation metrics matter for customer service?

The AI agent evaluation metrics that matter for customer service are correct resolution, answer correctness, policy adherence, action accuracy, escalation quality and tone, backed by repeat contact and customer feedback. Containment and deflection only tell you the customer did not reach a person, so they belong in the context column, not in the quality score.

Pair containment rate with a resolution check, see how to measure AI agent performance for comparing AI and human handling fairly, and put the criteria on an AI agent scorecard.

LLM-as-a-judge or human review: how do you calibrate them?

Use an LLM judge to score every conversation and human reviewers to calibrate it and to own the high-risk cases. Calibrate by having two reviewers and the judge score the same 100 to 200 conversations blind, then measure agreement for each criterion separately and fix the criterion wording wherever the judge and the humans disagree.

LLM-as-a-judge means a language model grades a conversation against your written criteria (our definition). The paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena found GPT-4 reached over 80% agreement with human preferences, the same level at which humans agree with each other. It also lists the judge’s weak spots: position, verbosity and self-enhancement bias, and limited reasoning.

A calibration routine that holds up:

• Score blind. Two reviewers and the judge score the same set separately. Reviewers agree first, then compare with the judge.
• Measure per criterion. An overall 90% can hide a tone criterion at 70%. For pass or fail criteria, track recall: of the real policy breaches, how many did the judge catch?
• Let the judge say “unknown”. Anthropic’s post recommends giving the judge a way out to reduce invented grades. Route unknowns to a person.
• Re-calibrate after every change to the scorecard, the judge’s model or the agent’s scope, and at least monthly.

Our one-week protocol to validate AI QA scoring turns this into a day-by-day plan with agreement thresholds.

How many conversations do you need to evaluate?

For pre-launch evals, 20 to 50 real failure cases is enough to start, growing to a few hundred as you automate more intents. For production evaluation of an AI agent, score every conversation if you can, because the failures that cost most are rare, and rare failures hide in samples. Keep human double-scoring at 100 to 200 conversations a month.

Illustration (not customer data). An AI agent handles 40,000 conversations a month. The QA team reviews a 1% sample: 400 conversations.

The chance of seeing none is (1 minus the rate) to the power of 400. Even two caught refund errors look like noise; scoring all 40,000 conversations shows 200 and 80 cases, grouped by intent.

Three rules of thumb:

• Margin of error. A 90% correct-resolution rate measured on 400 conversations is accurate to about ±3 points (95% confidence), so a drop to 88% is inside the noise. On 40,000 conversations the margin is about ±0.3 points.
• Zero failures is not zero risk. If you see no failures in n conversations, the true rate can still be as high as about 3 divided by n: showing a rate under 1% takes about 300 clean conversations, under 0.1% about 3,000.
• Calibration sets are small on purpose. 200 double-scored conversations with 85% agreement pin that agreement within about ±5 points: enough to find a badly worded criterion, not to rank reviewers.

The math is explained in our post on QA sampling statistics.

The AI agent evaluation framework (copy this table)

This framework lists what to evaluate, the method, the metric, an example threshold, the owner and the cadence for a customer service AI agent. The thresholds are examples to argue with, not industry benchmarks: set yours from your own baseline and from what a single failure costs you.

Why the consistency threshold is strict: a task that passes 98% of single runs passes all 5 runs about 90% of the time, and one at 95% only about 77% of the time. Customers get whichever run happens.

Who should own AI agent evaluation?

AI agent evaluation needs three owners: the AI or automation owner runs pre-launch evals and fixes prompts and tools, the QA lead runs production scoring and judge calibration, and the CX leader sets thresholds and decides whether the agent expands, holds or is restricted. Knowledge and policy owners fix the content the scores point to.

The common failure is evaluation owned only by the team that builds the agent: builders measure what they can change, while support leaders need to see what customers got. Put a support person in charge of the production numbers; the framework table above names an owner per metric. Evaluation is one part of a wider oversight model, covered in our guide to AI agent governance.

Which AI agent evaluation tools do support teams use?

Support teams use four kinds of AI agent evaluation tools: developer eval and observability platforms for pre-launch tests and traces, simulation tools that play the customer, the analytics built into the agent platform, and conversation QA platforms that score real conversations on a scorecard. Most enterprise teams end up with one tool per stage.

Kaizo sits in the last row. AutoPilot scores every resolved conversation against your own scorecard, and every score links to the evidence in the conversation. You test the scoring against your team’s historical ratings before it goes live, and reviewers keep the conversations that need judgment. Our page on QA for AI agents shows how it works inside Zendesk and Salesforce.

See it on your own conversations. Kaizo scores 100% of your support conversations and turns the findings into coaching. Book a demo.

When this level of evaluation is not for you

If your AI agent handles a few dozen conversations a week, skip the framework, the judge and the tool: read every transcript yourself for an hour a week against a short checklist (correct, allowed, right handoff, right tone). Start the full program when volume passes what one person can read, or when the agent starts taking actions such as refunds or account changes.

If what you need is a developer test suite in your release pipeline, that is a different tool; Kaizo’s QA platform scores conversations after they end and can run alongside it.

How to start AI agent evaluation this week

You can start AI agent evaluation this week without buying anything. In five days a support team can pick three intents, collect 30 real failures, write five yes or no criteria, score 100 recent conversations as a baseline and agree owners and thresholds. Everything else in the framework grows from there.

1. Monday: pick the highest-volume intent, the riskiest (usually refunds or account changes) and the one customers complain about most.
2. Tuesday: pull 10 bad conversations per intent from QA, complaints and escalations; write the correct outcome for each in one sentence.
3. Wednesday: write five criteria (correct answer, policy, action matches the system, escalation, tone), each a yes or no question with the evidence that proves it.
4. Thursday: two reviewers score 100 recent conversations blind. This is your baseline and first calibration set.
5. Friday: fill in the framework table with your names, baseline numbers and a date for the first monthly review.

Hand the 30 failures to your AI team as the first regression set: nothing ships until it passes.

Frequently asked questions

What is a good accuracy rate for an AI customer service agent?

There is no universal number; it depends on what one wrong answer costs. As an example threshold, aim for 98% or higher of conversations with no factual error and zero critical policy breaches on your regression set. Set the live threshold from your own baseline and investigate any drop of 3 points or more.

How often should you re-run AI agent evals?

Re-run the regression set before every prompt, model, tool or knowledge change, because each one can break something that worked. Score production conversations continuously, review the numbers weekly, and calibrate the judge monthly and after scorecard changes.

Should AI agents be evaluated on the same scorecard as human agents?

Use shared criteria where they apply to both, such as a correct answer and the right tone, so results stay comparable. Then add criteria only an AI agent needs: grounding in your content, action accuracy, scope and escalation triggers. One standard with AI-specific checks works better than two unrelated scorecards.

What is the difference between pass@k and pass^k?

pass@k is the chance that at least one of k runs succeeds, which suits tools where one good answer is enough. pass^k is the chance that all k runs succeed. Customer service needs pass^k, because every customer gets one run and expects the right answer every time.

Is AI agent evaluation the same as AI agent monitoring?

No. Monitoring watches live volume, errors and alerts so you notice a problem quickly. Evaluation judges whether each answer was correct, allowed and well handled, before launch on test cases and after launch on real conversations. Production evaluation feeds monitoring, and every failure it finds becomes a new test case.