To QA AI voice agents, you score the calls your voice bot handles against a written standard, after each call ends, so you know whether it understood the caller, gave a correct and allowed answer, and handed over to a person when it should have. You need a scorecard with voice-specific checks, a transcript of every call, and a way to score all of them rather than a sample.
In short
- A voice agent fails in ways a chatbot does not (misheard numbers, talking over the caller, lost transfers), so its scorecard adds checks for recognition, read-back, escalation and disclosure.
- A small random sample of calls tends to miss a voice agent’s failures, because they cluster on one intent and repeat.
- Six steps cover the job: risky intents first, the scorecard, transcripts, scoring every call, the handoff, and re-scoring after each fix.
- Since August 2, 2026, the EU AI Act requires that callers are told they are talking to an AI, and US call recording rules still apply.
- Testing before launch and QA of live calls are different stages, and you need both.
This guide is the voice version of our general guide on how to QA AI agents. For the product side, the page on AI agent QA shows how Kaizo scores AI agent conversations on the same scorecard as your team’s.
What is different about QA for AI voice agents?
QA for an AI voice agent checks what the bot heard as well as what it said, because a voice agent turns speech into text before it answers. An error in that first step changes everything after it. A chatbot reads what the customer typed; a voice agent guesses.
That adds three kinds of failure that text QA never sees:
- Recognition errors. The bot hears “fifteen” as “fifty”, gets one digit of an order number wrong, or mangles a surname. The answer that follows can be perfectly grounded in your content and still be about the wrong order.
- Conversation timing. The bot answers before the caller has finished, leaves long silences, or keeps talking when the caller tries to interrupt. On a chat transcript none of this exists.
- The transfer. A voice handoff is a live transfer. If the context does not travel with the call, the caller repeats everything to a person who already sounds unprepared.
The rest is the same as any AI agent: the answer has to be correct, come from your own content, stay within policy, and escalate on the right trigger. We cover those shared checks in the AI agent scorecard, and the transfer in detail in scoring the human-AI handoff.
Which failures should a voice agent scorecard catch?
A voice agent scorecard should catch wrong answers, invented policy, misheard details, unconfirmed actions, missed escalations and a missing AI disclosure. For each check, it should also say whether a transcript is enough or the audio is needed, because that decides how you review.
Here is a scorecard you can copy and adapt. The first four rows are the ones your human agents are already scored on, which keeps the two sets of results comparable.
| Criterion | What it checks | Scored for | Transcript enough? |
|---|---|---|---|
| Correct answer | The answer solves the caller’s question | Human and AI agents | Yes |
| Policy limits | No invented refund, fee waiver or exception | Human and AI agents | Yes |
| Identity verified | The required checks ran before account details were shared | Human and AI agents | Yes |
| Tone | It sounds like your brand, not like a script | Human and AI agents | Partly: pace and talking over the caller need the audio |
| Read-back of key details | Numbers, dates, names and amounts were confirmed back to the caller | AI agents | Yes |
| Recognition | What the bot acted on matches what the caller said | AI agents | No: compare the audio with the transcript on failed calls |
| Grounded in your content | Every fact comes from your knowledge base | AI agents | Yes |
| Escalation | It offered a person on the right trigger, including “agent” or “human” said twice | AI agents | Yes |
| Handoff | The context went with the transfer and the person picked it up | AI and human agents | Yes, if both legs of the call are in one record |
| AI disclosure | The caller was told at the start that they were speaking with an AI | AI agents | Yes |
The read-back row is the cheapest fix with the largest effect. A voice agent that repeats “So that’s order 4-8-1-7, a refund of 15 dollars, is that right?” turns a silent recognition error into a correction the caller makes for you. Score it on every call that changes an account, takes a payment or books something.
Why does a sample of calls miss voice bot failures?
A random sample misses voice bot failures because they concentrate on one intent and repeat. A fault that ruins a large share of one kind of call is still a small share of all calls. The math below shows how often a typical sample sees nothing.
Worked example (illustrative numbers, not Kaizo data). Say your voice agent handles 12,000 calls a month and your team reviews a random sample of one call in 50, which is 240 calls. One call in 20 is a refund call, so 600 calls.
After a policy change, the bot gives the wrong refund window on one refund call in ten: 60 calls, or one call in 200.
| Value | |
|---|---|
| Calls per month | 12,000 |
| Calls reviewed (2% sample) | 240 |
| Refund calls with the wrong answer | 60 (0.5% of all calls) |
| Expected wrong refund calls in the sample | 1.2 |
| Chance the sample contains none of them | about 30% |
| Chance it contains one at most | about 66% |
The last two rows follow from the binomial formula: the chance of drawing none is 0.995 to the power of 240, about 0.30.
In roughly two months out of three your reviewers see one bad refund call or none, which looks like noise. Scoring every call turns the same fault into 60 failed calls on one criterion, with the transcripts attached.
The general version of this argument, with the formulas, is in why a 3% QA sample cannot support agent-level decisions.
How to QA AI voice agents in six steps
To QA AI voice agents, start with the costly intents, write a scorecard with voice checks, get a transcript of every call, score all calls, check the transfers, and re-score after every fix.
Step 1: List the risky intents
Pull the intents your voice agent handles and mark the ones where a mistake costs money or trust: refunds, payments, cancellations, address changes, appointment bookings and anything that touches identity. These get scored first and read most often.
Step 2: Write the scorecard
Start from the table above. Keep the criteria your human agents share with the bot, add the voice checks, and write each criterion as a yes or no question an AI or a reviewer can answer from the call. Our guide to writing a QA rubric an AI can score shows how to phrase them.
Step 3: Get a transcript of every call
Scoring needs text. Check that every voice agent call ends up with a recording and a transcript in the system where your team’s calls live, and that the transcript marks who said what. If bot calls sit in a separate vendor console, your human and AI results will never be on one report.
Step 4: Score every call, not a sample
Score all voice agent calls against the scorecard, with a comment on each score that points to the line behind it.
Use reviewers for the calls that fail a high-weight criterion and for a set of calls that passed, to check the scoring itself. Before you rely on automated scores, test them against your own past ratings, as described in how to validate AI QA scoring.
Step 5: Check the transfers
Filter for calls the voice agent transferred and calls where the caller asked for a person. Score whether the bot escalated on the right trigger, whether the context reached the human agent, and whether the caller had to repeat themselves.
Then look at the calls that ended without a transfer and without a resolution: a caller who hangs up is often counted as contained.
Step 6: Fix, then re-score
Trace each failure to its cause: a knowledge base article, a prompt, a missing read-back, an escalation rule or a speech recognition problem with one kind of detail. Fix it, then watch that criterion on the same intent for the next week.
A fix to one answer can break another, which you only see if you keep scoring all calls.
See it in a live demo. Kaizo scores calls, chats and emails against one scorecard, and scores AI agent conversations on the same scorecard as your team’s. Let’s talk.
What does the law ask of an AI voice agent?
In the EU, a voice agent must make clear to callers that they are talking to an AI, from the start of the call, since August 2, 2026. In the US, the usual call recording rules apply, and outbound AI voice calls fall under the robocall rules.
| Rule | What it means for a voice agent | Source |
|---|---|---|
| EU AI Act, Article 50(1) | Providers must design AI systems that interact directly with people so those people are informed they are interacting with an AI, unless that is obvious. Applies from August 2, 2026. | Regulation (EU) 2024/1689 on EUR-Lex |
| Commission guidance on Article 50 | Notice is due from the start of the first interaction, clear and accessible; chatbots and AI agents are named as examples. | European Commission FAQ on Article 50 |
| US federal recording law | Recording is allowed with one party’s consent under 18 U.S.C. § 2511(2)(d). | Cornell LII |
| US state recording laws | Some states, California among them, require the consent of all parties. | California Penal Code § 632 |
| US outbound calls | The FCC ruled on February 8, 2024 that AI-generated voices count as “artificial” under the TCPA, so the consent rules for robocalls apply. | FCC announcement |
This table is a summary, not legal advice.
For QA, the practical consequence is simple: make the AI disclosure and the recording notice criteria on the scorecard, so you can show they were given on every call rather than assume it. Guidance and national rules can still change, so check the current position with your counsel before you launch in a new market.
Testing before launch vs QA after the call
Testing before launch checks the voice agent against scripted calls in a controlled setup; QA after the call checks what it actually did with real callers. They catch different problems, and a voice agent needs both.
| Pre-launch testing | QA of live calls | |
|---|---|---|
| Input | Scripted or simulated test calls | Real calls from real customers |
| When | Before release and after each change to prompts or flows | After every call ends |
| Catches | Broken flows, prompt regressions, latency under load | Real accents and noise, new questions, policy drift, failed transfers |
| Who owns it | The team that builds the bot | QA and support operations |
| Typical tool | A test suite or simulation tool | A QA platform with a scorecard |
Kaizo sits in the second column. It scores conversations once they have ended, and its site is direct about it: testing a voice agent before launch with scripted test calls is a different stage from QA of live conversations. If you need live call listening or real-time prompts during the call, that is real-time agent assist, another category again.
How to start QA on your voice agent
Start this week with one intent: pick the riskiest one your voice agent handles, write ten scorecard criteria from the table above, and score every call on that intent for two weeks. You will learn more from those calls than from a month of sampling across all intents.
Then widen the scope intent by intent. To see how one scorecard covers human agents, chatbots and calls, go to the page on QA for AI agents.
Frequently asked questions
How do you test an AI voice agent before it goes live?
Run scripted test calls that cover your main and risky intents, plus the hard cases: background noise, accents, interruptions, fast numbers and callers who ask for a person. Check the flows, answers and response time. Repeat the same test set after each change to the prompt or the flow, because a fix in one place can break another.
What metrics matter for an AI voice agent?
Track outcome metrics such as containment, transfer rate and repeat calls, but judge quality on a scorecard: correct answer, policy limits, read-back of key details, escalation and handoff. Containment alone counts a caller who gave up and hung up as a success. Our guide on containment rate explains how to read it without fooling yourself.
Can you score voice bot calls and human calls on the same scorecard?
Yes, for the criteria that apply to both, such as correct answer, policy limits, identity checks and tone. That keeps the results comparable, so you can see where the bot is stronger or weaker than your team. The voice agent then gets extra criteria for recognition, read-back, escalation and the AI disclosure.
Does Kaizo score phone calls?
Yes. Kaizo scores calls against the same scorecard as chat and email, and every plan includes AI summaries and on-demand transcription. Kaizo reviews a conversation once it has ended and the case resolves, so it is a QA tool for live calls after the fact, not a live monitoring tool.
How many voice agent calls should you review by hand?
Score every call automatically if you can, then review by hand the calls that fail a high-weight criterion and a small set that passed, to check the scoring. If you review only by hand, read every call on your riskiest intent first, because voice agent failures cluster on one intent and repeat.
