You catch AI agent hallucinations by checking every answer the bot gives against your own content and policy, not by waiting for complaints. Score each AI agent conversation on grounding and policy criteria, read the money topics first, and trace each confirmed hallucination to the article or rule behind it, so the second customer never gets the same wrong answer.
In short
- An AI agent hallucination is a confident answer your content does not support: an invented policy, a wrong number, an action the bot never took.
- The wrong refund answer is the one to fear, because customers act on it and the company stays responsible for it.
- Samples miss hallucinations: in a worked example, a 2% sample has a 23% chance of showing none of 72 wrong refund answers.
- A six-step routine catches them: list the risky claims, write checkable criteria, score against your content, score everything, triage by severity, fix at the source.
- Post-conversation QA cannot stop the first wrong answer. It stops the next hundred, which is where the cost is.
If you run AI agents in front of customers, our page on AI agent QA shows how Kaizo scores every AI agent conversation against your own criteria, including checks for invented facts and invented policy. This guide is the method, whatever tool you use.
What is an AI agent hallucination in customer service?
An AI agent hallucination is an answer that sounds certain but is false or not supported by your content. In support, it usually takes the form of a policy, price, promise or fact the customer then acts on.
The NIST Generative AI Profile (July 2024) calls this confabulation: “the production of confidently stated but erroneous or false content (known colloquially as ‘hallucinations’ or ‘fabrications’) by which users may be misled or deceived.” The last part matters most in support. A customer cannot tell a confident wrong answer from a confident right one.
In customer service, hallucinations fall into six types. Each one needs a different check:
| Type | Example | How to check it |
|---|---|---|
| Invented policy | “You can request the refund within 60 days” when the policy says 30 | Compare against the policy article the bot should have used |
| Wrong number | An old price, a wrong fee, a wrong delivery time | Compare every number in the answer with the source |
| Invented action | “I’ve processed your refund” when the bot cannot process refunds | Compare the claim with the actions the bot is allowed to take |
| Invented exception | A fee waived or a deadline extended that nobody approved | Check against the list of exceptions only a human may grant |
| Wrong product fact | A feature, plan limit or compatibility claim that is not true | Compare with the product documentation |
| Invented source | A help article, form or link that does not exist | Open the link or search the help center for the title |
A hallucination is one kind of silent failure, the errors a dashboard counts as wins. Our taxonomy of silent AI agent failures covers the others, such as missed escalations and false resolutions.
Why is the wrong refund answer the one to fear?
The wrong refund answer is the one to fear because the customer acts on it, and your company remains responsible for what its bot said. It turns a support error into a cost, a complaint or a legal claim.
The best-known case is Moffatt v. Air Canada (February 2024). The airline’s website chatbot told a customer that a bereavement fare could be requested after travel. The airline’s policy did not allow that, and the chatbot’s own link pointed to a page that said so.
British Columbia’s Civil Resolution Tribunal rejected the argument that the chatbot was responsible for its own actions. It found that Air Canada “did not take reasonable care to ensure its chatbot was accurate” and ordered it to pay CA$812.02. One conversation is a small amount. The same answer repeated to every customer who asked the same question is not.
Money topics are also where hallucinations are most likely to look helpful. A bot that invents a generous refund window produces a happy customer, a resolved conversation and a good satisfaction score, right up to the moment the refund is refused.
Why do samples miss AI agent hallucinations?
Samples miss AI agent hallucinations because a hallucination on one topic is rare across all conversations but repeats within that topic. A small random sample sees too few conversations on the topic to show the pattern.
Here is a worked example with numbers we chose to show the arithmetic, not a customer’s data:
| Step | Number |
|---|---|
| AI agent conversations in a month | 40,000 |
| Conversations about refunds (6%) | 2,400 |
| Refund answers with an invented condition (3% of refund conversations) | 72 |
| Random QA sample (2% of all conversations) | 800 |
| Refund conversations you can expect in that sample | 48 |
| Wrong refund answers you can expect in that sample | 1.44 |
| Chance the sample contains none of them | 23% |
| Chance it contains none or one | 58% |
The two chances come from the binomial distribution: 0.97 to the power of 48 is 0.232, and adding the chance of exactly one gives 0.576. So in more than half of the months, your reviewers see at most one wrong refund answer. One looks like an outlier, so nobody fixes the article behind it.
Score every conversation and you see all 72, grouped on one topic, within the week. That is why our guide on how to score 100% of conversations matters more for bots than for people: a person’s errors scatter, while a bot’s repeat.
How to catch AI agent hallucinations: a six-step routine
Catch AI agent hallucinations with six steps: list the risky claims, write checkable criteria, score against your own content, score every conversation, triage by severity and fix each one at its source. Steps 1 to 3 are set up once.
Step 1: List the claims that cost money if wrong
Start with the topics where the customer acts on the answer: refunds, cancellations, fees, prices, delivery dates, eligibility and account changes. For each topic, write down the claims the bot makes, such as “refund window”, “cancellation fee” or “which plans include X”. This list becomes your scorecard’s focus and your first review queue.
Step 2: Write criteria a reviewer can check against the transcript
A criterion like “the answer is accurate” is too vague to score the same way twice. Write it so a reviewer, or an automated scorer, can point at the line that passes or fails it:
| Criterion | Pass when | Fail when |
|---|---|---|
| Grounded in our content | Every fact in the answer appears in a help article or policy we publish | The answer contains a fact, number or condition our content does not state |
| Policy limits | Refunds, credits and exceptions stay within the written policy | The bot offers or promises anything the policy does not allow |
| Actions claimed | The bot only says it did what it can actually do | The bot says it processed, cancelled or changed something it cannot |
| Uncertainty | The bot says it does not know, or hands over, when the content has no answer | The bot fills the gap with a guess |
Our guide to writing a QA rubric an AI can score shows how to phrase the rest of the scorecard the same way.
Step 3: Score against your own content
A grounding check is only as good as the content it checks against. Connect the scorer to the same help center and policy articles the bot answers from, so “not supported by our content” has a precise meaning.
In Kaizo you write the criteria in your own words and connect your knowledge base, so the AI judges each answer against your content.
Step 4: Score every AI agent conversation
Score all of them, because the worked example above shows what a sample hides. Kaizo’s AutoPilot scores each conversation against your criteria when it resolves and attaches a comment that explains the score. You can scope which queues, channels or teams it scores, so the money topics are covered from day one.
Step 5: Triage by severity, not by date
Not every hallucination needs the same response. A severity rule keeps the team on the ones that cost money:
| Severity | What it looks like | Response |
|---|---|---|
| Critical | Invented refund, fee waiver, legal or safety claim, a promise the company must honor or withdraw | Same day: contact the affected customers, pull the topic from the bot or force a handover until fixed |
| High | Wrong number or condition on a money topic, invented action | Within 2 working days: fix the source article, re-score the topic |
| Medium | Wrong product fact with no cost attached, invented link | In the weekly content fix list |
| Low | Harmless detail, such as a wrong store opening example | Batch with the monthly content review |
Step 6: Trace each hallucination to its source and re-score
Most hallucinations come from something you can fix: a help article that is outdated, two articles that disagree, a condition buried at the bottom of a page, or a topic with no article at all.
Fix the source, then re-score that topic over the following week to confirm the fix held. Kaizo’s root-cause analysis groups the failures behind a dip, so ten bad answers from one article become one fix.
Which signals point to a hallucination?
Customer behavior after the conversation is the cheapest hallucination signal: repeat contacts, disputes and reversals by human agents. Use these signals to choose which conversations to read first.
- The customer comes back within 7 days on the same topic, often on another channel.
- The customer quotes the bot to a human agent: “your assistant told me”.
- A human agent withdraws or corrects something the bot promised.
- A refund, credit or exception request follows a bot conversation on that topic.
- The bot’s answer contains a number, date or link that does not appear in the source article.
- The customer pushes back (“that’s not what your website says”) and the bot repeats itself.
Each signal is also a scorecard criterion or a filter. The handoff signals in particular belong on the scorecard, as our guide on scoring the human-to-AI handoff explains.
What post-conversation QA cannot do
Post-conversation QA cannot stop the first wrong answer, because it scores conversations once they end. It stops the same wrong answer from reaching the next customers, and it finds who already received it.
Kaizo scores conversations after they end, in the helpdesk where they live. It does not monitor the bot live or block an answer before it is sent.
Guardrails that run while the bot answers, and pre-launch tests such as prompt regression tests in a developer pipeline, belong with the team that builds the agent. Those are different stages and run alongside QA.
What post-conversation scoring adds is the part the other stages cannot see: what the bot actually told real customers, on every conversation, with the transcript behind each finding. That is also the evidence you need to contact the customers who got a wrong answer before they find out on their own.
How to start catching hallucinations this month
Start with one money topic, four criteria and every conversation on that topic for two weeks. That is enough to find your first source to fix and to measure your hallucination rate.
- Pick the topic where a wrong answer costs most, usually refunds or cancellations.
- Add the four criteria from Step 2 to the bot’s scorecard.
- Score every AI agent conversation on that topic for two weeks.
- Count the conversations that fail a grounding or policy criterion, divided by all conversations on the topic. That is your hallucination rate for the topic.
- Fix the top source, re-score for a week, and compare.
See it in a live demo. Kaizo scores every AI agent conversation against your own content and criteria, and links each failed criterion to the line that failed it. Let’s talk.
Frequently asked questions
Can you stop an AI chatbot from hallucinating completely?
No current method guarantees it. Grounding the bot in your content, limiting its topics and forcing a handover when it lacks an answer all reduce hallucinations. What you can guarantee is that you find them: score every conversation against your content, so a wrong answer is caught on the first conversations it appears in, not after a complaint.
Does a knowledge base stop an AI agent from making things up?
It reduces the problem but does not remove it. A bot that answers from your knowledge base can still combine two articles wrongly, use an outdated one, or fill a gap where no article exists. That is why the grounding check compares each answer with the source, and why an outdated or missing article is the most common root cause.
How do you measure an AI agent’s hallucination rate?
Count the conversations that fail a grounding or policy criterion and divide by all scored conversations, per topic. Measure per topic rather than overall, because a low overall rate can hide a high rate on refunds. Track it weekly while you fix sources, then monthly once it is stable.
Who is responsible when a chatbot gives a customer wrong information?
In the Air Canada case, the company was. In February 2024, a British Columbia tribunal ruled that the airline was responsible for all the information on its website, including its chatbot’s answers, and ordered it to pay CA$812.02. That is one small claims decision in one province, not a general rule, so check the law where you operate.
How fast should you fix a hallucination once you find it?
Fix it by severity. Pull a critical topic from the bot or force a handover the same day, and contact the customers who got the wrong answer. Fix high-severity source articles within two working days, then re-score the topic for a week to confirm the fix held. Lower severities can go into a weekly content fix list.
