AI agent testing is how a support team proves an AI agent still behaves before a change reaches customers. You build test cases from real resolved conversations and past failures, rerun them before every prompt, policy, knowledge or model change, simulate difficult customers, check the actions the agent took, and ship only when a written release gate passes.
In short
- An agent that takes actions needs tests on what it did (refund issued, limit respected, case handed over), not only on what it said.
- The best test cases come from your own resolved conversations and past failures.
- Rerun the full regression set before every prompt, policy, knowledge, model or tool change.
- Run each critical case several times: an agent that passes once can fail the same case on the next run.
- Write the release gate down before the test run, so nobody argues thresholds after seeing results.
This guide is the pre-release part of our wider guide to AI agent evaluation.
What is AI agent testing, and how is it different from chatbot testing?
AI agent testing checks an agent that reasons over several turns and takes actions in your systems, such as issuing refunds, changing orders or updating accounts. Chatbot testing mostly checks answers and conversation flows. The difference is consequence: a wrong chatbot answer misleads one customer, while a wrong agent action changes real records.
If your bot only answers and hands over, see chatbot testing. This article covers what changes once the agent can act.
| What you test | Chatbot | AI agent that takes actions |
|---|---|---|
| The path | Scripted flow, predictable | Varies run to run; several valid paths |
| Actions | Few or none | Tool calls with real effects: refunds, cancellations, address changes |
| Consistency | Mostly deterministic | Same input can produce a different result on the next run |
| Handover | Button or keyword | Judgment call the agent makes mid-conversation |
Consistency is the hard part. In the τ-bench study, where a language model plays the customer and the agent works with domain-specific tools and policy guidelines, even strong function-calling agents succeeded on fewer than 50% of tasks and passed the same retail task in all 8 trials less than 25% of the time. Success was judged by the database state at the end of the conversation, not by the reply.
What should an AI agent test case include?
An AI agent test case needs six core fields: the scenario, the customer persona, the expected outcome, the actions the agent is allowed to take, a pass rule a second person would apply the same way, and the source conversation it came from. Without the last two, test results turn into opinions.
Here is the template: the six core fields plus four that keep the set maintainable. Copy it into a spreadsheet, one row per case.
| Field | What to write | Example |
|---|---|---|
| Case ID and tags | A stable ID plus tags for intent, risk and channel | RF-014 · refund · high risk · chat |
| Scenario | What the customer wants and the facts the agent can look up | Customer received a damaged blender, order 31 days old, paid 89 USD |
| Customer persona | Tone, knowledge, behavior over the turns | Polite at first, pushes back after the first answer, asks for “just the refund” |
| Expected outcome | The end state in your systems, plus what the customer must be told | No automatic refund (window is 30 days); replacement offered or case handed to a person with order details |
| Allowed actions | Which tools the agent may call, with limits | Look up order: yes. Create replacement: yes. Issue refund: no (outside window). Hand over: yes |
| Pass rule | Binary and checkable by someone else | Pass if no refund is issued, the customer is not promised one, and either a replacement is created or the handover note names the order and the damage |
| Severity | What a failure means | Critical: money or policy. Major: wrong answer, no harm. Minor: tone |
| Trials | How many runs must pass | Critical: 3 of 3. Others: 2 of 3 |
| Source conversation | Link to the real conversation it came from | Ticket link, plus the date it was added and why |
| Owner and last review | Who keeps the case current | Refund policy owner, reviewed when the policy changes |
Two rules keep the template useful. First, the expected outcome describes a state, not a sentence: Anthropic’s engineering guide to agent evals makes the same point with a flight-booking example, where what counts is whether a reservation exists in the database, not whether the agent says it booked one. Second, every case points to the conversation that justifies it, so when a policy changes you can find and update the cases it touches.
How do you build a golden test set from real conversations?
Build the golden test set from your own traffic: resolved conversations that show the agent at its best, past failures you never want to see again, escalations, and the edge cases of each policy. Start with 20 to 50 cases and grow it every week, rather than waiting for a perfect set of hundreds.
Anthropic’s guide says 20 to 50 simple tasks drawn from real failures is a great start, and recommends converting user-reported failures into test cases so the suite reflects actual usage.
A balanced first set of 50 cases might look like this (an illustration; shift the weights to your own volume and risk):
| Bucket | Cases | Where they come from |
|---|---|---|
| Top intents, happy path | 15 | Your 10 to 15 highest-volume contact reasons, one clean resolved conversation each |
| Action cases | 10 | Refunds, cancellations, plan changes, address updates: inside and just outside each limit |
| Past failures | 10 | Conversations where the agent was wrong, reopened, or a customer complained |
| Handover cases | 6 | Cases the agent must pass to a person: legal threats, vulnerable customers, fraud signals |
| Unknown-answer cases | 4 | Questions your knowledge base does not answer; the agent must say so, not invent |
| Adversarial cases | 5 | Attempts to talk the agent past a limit or out of its instructions |
Keep the passing cases in the set: a fix to one flow can quietly break another, and only cases that used to pass will show it. Strip personal data from every source conversation before it becomes a test case.
How do you run regression tests before every AI agent change?
Run regression tests by replaying the full golden set against the changed agent before it goes live, with the same pass rules as last time, and comparing the results with the current version. Any change counts: prompts, policies, knowledge articles, model versions, tools and routing rules all alter behavior. An edited help article can change refund answers as much as a prompt edit.
| Change | What usually breaks | Minimum rerun |
|---|---|---|
| Prompt or instructions | Tone, scope, handover timing | Full set |
| Policy (refund window, limits) | Action cases at the edge of the old and new limit | Full set, plus new edge cases for the new limit |
| Knowledge article added or edited | Answers that cite it; neighboring topics | Cases tagged with that topic, then the full set before release |
| Model version | Anything, often subtly | Full set, critical cases with extra trials |
| New tool or permission | Actions, wrong tool chosen, actions without confirmation | Action and adversarial cases, then full set |
| Routing or handover rule | Cases that should or should not reach a person | Handover cases, then full set |
Run critical cases more than once. Agents are not deterministic, so a single pass proves little; the τ-bench authors proposed pass^k, the chance that all k runs succeed, for this reason. The arithmetic (illustration): a case the agent passes 90% of the time passes 3 runs in a row only 73% of the time (0.9 × 0.9 × 0.9), and 8 runs in a row only 43% of the time.
Regression cases should pass at, or very near, 100%. Anthropic’s guide makes the same split: regression evals should have a nearly 100% pass rate, while capability evals for new skills may start low.
How do you simulate customers to test an AI agent?
Simulate customers by having a second model play the customer from a written persona, over several turns, against the agent you are testing. Give each persona a goal, facts, a temper and a breaking point, and add adversarial personas that push the agent past its limits.
Single questions test answers. Multi-turn simulation, the design τ-bench used, tests what goes wrong in real conversations: the customer who changes the request halfway or keeps pushing for a refund the policy does not allow.
Write five to eight personas and reuse them across scenarios:
- The clear asker: one request, all details given. Your baseline.
- The vague one: missing order number, wrong product name. Tests clarifying questions.
- The switcher: starts with delivery, ends with a cancellation.
- The pusher: rejects the first answer three times and asks for an exception. Tests limits under pressure.
- The upset customer: threatens to leave or complain publicly. Tests tone and handover.
- The vulnerable customer: mentions bereavement, illness or money trouble. Tests your handover rules for these cases.
- The manipulator: claims to be staff, pastes “ignore your instructions”, or hides instructions in an attachment. Tests resistance to what OWASP calls prompt injection.
A persona only finds the problems you thought to write into it, so use simulation to widen the set and real conversations to keep it honest. Voice agents add accents, interruptions and line quality; see QA for AI voice agents.
How do you test an AI agent’s actions and tool calls?
Test actions by checking the end state in your systems after the conversation, not the agent’s description of what it did. Did the refund record exist, for the right amount, on the right order? Did the agent stay inside its limits, ask for confirmation where required, and call no tool it was not allowed to?
A reply saying “I have processed your refund” proves nothing. Check four things per action case:
- Right action. The expected tool was called, and no unexpected one.
- Right arguments. Correct order, customer, amount and reason.
- Inside limits. Amount under the cap, permission held, approval asked where your policy requires a person.
- Truthful report. What the agent told the customer matches what actually happened in the system.
OWASP lists excessive agency among its top risks for LLM applications: damaging actions enabled by too much functionality, too many permissions or too much autonomy. Its mitigations (minimal extensions, minimal permissions, user approval) double as a test list. Write at least one case per limit where the customer asks for just over it.
Run action tests against a sandbox or test accounts, never live records. Salesforce’s training for its Agentforce Testing Center warns that testing agents can modify CRM data and says to use it only in a sandbox.
Handover is an action too: test that the agent hands over the cases it should, with a note that lets the person continue (escalation handling quality).
How do production failures become new test cases?
Production failures become test cases through a weekly routine: find the conversations where the AI agent went wrong, confirm the failure, strip personal data, write the case with the template above, and add it to the golden set. The next change cannot ship until that case passes.
The hard part is finding the failures. A conversation the agent “resolved” can still contain a wrong answer, an invented policy or a missed handover, and dashboards count it as a success (silent AI agent failures). AI agent monitoring covers what to watch day to day, and how to QA AI agents shows how to build the scorecard that catches them.
This is where QA of live conversations and pre-release testing meet. Kaizo’s AutoPilot scores every resolved AI agent conversation against your own scorecard and links each score to the evidence, which gives you confirmed failures, each with a source conversation for the template. After the fix ships, scoring the next weeks of live conversations shows whether it held.
See where your AI agent fails today. Kaizo scores AI agent conversations on the same scorecard as your team and links every score to the lines behind it. Book a demo.
If you score with an AI judge, check the judge too. Our guide on how to validate AI QA scoring shows how to compare automated scores with your reviewers before you trust them.
What should an AI agent release gate check?
An AI agent release gate is a short written list of conditions a change must meet before it reaches customers, agreed before the test run. It should block on any critical failure, require near-perfect regression results, cover every action and limit, and name who signs off and how to roll back. Copy it and set your own numbers.
Before the run
- The change is written down: what, why, who asked.
- New cases for the change itself (a new limit gets cases just inside and just outside it).
- Every failure confirmed in production since the last release is in the set.
- Test accounts and sandbox ready; no live customer records in reach.
The run
- Full golden set run against the changed agent and the current agent, same pass rules.
- Critical cases (money, data, legal, vulnerable customers) run at least 3 times each.
- Simulated personas run, including the manipulator and the pusher.
Pass conditions
- Critical cases: 100% pass, every trial. One failure blocks the release.
- Regression cases: at least 98% pass, and no case that passed last release now fails without an explanation.
- Action cases: no action outside the allowed list, no amount over the limit, no claim of an action that did not happen.
- Handover cases: every must-hand-over case reached a person with a usable note.
- Unknown-answer cases: no invented answers.
Sign-off and after release
- Named owner signs: support or CX lead for policy and tone, AI owner for the technical side.
- Rollback is ready: the previous prompt, knowledge and model version can be restored the same day.
- The first days of live conversations after release are reviewed more closely than usual.
Worked example (an illustration with invented numbers). A team shortens its refund window from 30 to 14 days. Its golden set has 120 cases: 40 critical, 80 others. Critical cases run 3 times, others once, so the run has 200 trials. The result: 196 of 200 trials pass, a 98% pass rate that looks healthy. But all 4 failures sit in two critical cases. In one, the agent still offered a refund at day 20 in one of three trials. In the other, the pusher persona talked it into a “goodwill” refund in all three. The gate blocks the release. The fix goes into the prompt and a knowledge article, two new critical cases are added for day 14 and day 15, and the rerun passes 206 of 206 trials. An average would have shipped the bug; the gate did not.
What tools do you need for AI agent testing?
You need three kinds of tools: a place to keep and version test cases, a way to run them against the agent (often with simulated customers), and a way to score results against your pass rules.
- A spreadsheet and manual runs are enough under about 50 cases and a few changes a month.
- A platform testing feature suits one platform with standard flows. Salesforce’s Agentforce Testing Center, for example, generates 20 test cases by default, and you set how many are positive and negative.
- Testing and simulation tools add simulated customers, repeated trials and release checks in the deployment pipeline.
- QA of live conversations scores what really happened after release and feeds confirmed failures back into the set.
Whatever you choose, keep the cases in a format your support team can read and edit, or policy owners stop updating them.
When is a full testing program not for you?
A full program is overkill if your AI agent handles a few dozen conversations a week, only answers questions, and cannot take actions. In that case, read the transcripts weekly and keep a short list of 20 cases you rerun by hand after each change.
Nor does QA of live conversations replace it: scoring conversations after they end does not block a change before release. You need both, tests before the change and scoring of every AI agent conversation after it.
How to start this week
You can have a first golden set, a written release gate and a baseline run within five working days, with an export of recent conversations, the template above and a test environment.
- Monday: export 30 resolved AI agent conversations and 20 that went wrong. Remove personal data.
- Tuesday: write them into the template. Mark which are critical.
- Wednesday: write the release gate. Get the support lead and the AI owner to agree the numbers.
- Thursday: run the set against the current agent, three trials for critical cases. This is your baseline; expect some failures.
- Friday: agree the rule that no prompt, policy, knowledge or model change ships without a rerun, and set a weekly slot to add new failures to the set.
See AI agent governance for who owns the sign-off.
Frequently asked questions
What is AI automation in testing?
AI automation in testing usually means using AI to test software, for example agents that write and run test scripts for a web app. This article covers a different topic: proving that a customer-facing AI agent answers and acts correctly before each change goes live.
Do AI agents have tools?
Yes. An AI agent’s tools are the functions it can call, such as looking up an order or issuing a refund. They are why agent tests must check the end state in your systems, not only the reply.
What is an AI tester?
Either a person who tests AI systems or an AI that tests other software. In customer service it is usually a QA lead or AI owner who writes test cases from real conversations and decides whether a change passes the release gate.
How many test cases does an AI agent need?
Start with 20 to 50 cases drawn from real conversations and past failures, then add every confirmed production failure. Coverage of your top intents, actions and limits matters more than the raw count.
How often should you run AI agent regression tests?
Run the full set before every change to prompts, policies, knowledge, models, tools or routing, and weekly even when nothing changed, because model providers and connected systems change too.
