Customer journey QA reviews everything a customer went through to get one issue solved: every bot, AI agent, chat, call and human contact about that issue within a set window. Instead of asking whether each conversation was good, it asks whether the journey ended with the problem solved, with no loops, lost context or ignored requests for a person.
In short
- Conversation QA scores one contact. Journey QA scores the whole path a customer took to solve one issue, across bots, AI agents, channels and people.
- Most AI-era failures sit between conversations: an ignored “talk to a human”, a chat and phone loop, lost context, a repeat contact, a customer who silently gives up.
- Define a journey as the same customer with the same issue within a fixed window, 7 days by default.
- Track journeys resolved correctly, contacts per journey, handoff failures, context lost, human requests honored, silent abandonment and time to resolution.
- Review whole journeys on a journey scorecard, sample them on risk signals, and give every failed journey one owner who fixes the cause.
What is customer journey QA?
Customer journey QA is quality assurance where the unit of review is the customer’s issue, not the single conversation. A reviewer, or an automated check, follows one customer and one issue across every touchpoint (bot, AI agent, chat, email, phone, human agent) within a set window and judges whether the journey solved the problem correctly and without needless effort.
Traditional QA reads one ticket and scores the agent who handled it. With AI agents in front, one issue often becomes three or four contacts: a bot conversation, an escalation to chat, a phone call, a follow-up email. Each contact can pass its own scorecard while the customer spends three days getting nowhere.
“Customer journey AI agents” often means marketing tools, such as Microsoft’s Journey Creation Agent, which builds email, SMS and push journeys from a plain-language goal. This guide is about support: checking the journeys customers actually take when they need help.
Why does scoring single conversations miss where AI-era support fails?
Scoring single conversations misses AI-era failures because those failures happen between conversations. A bot that ignores “talk to a human”, a chat that sends the customer to the phone line, a handoff that drops the order number or a customer who quietly gives up all leave each conversation looking acceptable. Only the full journey shows the damage.
Customers expect this. In a Gartner survey of 5,728 customers (December 2023), 64% said they would prefer that companies did not use AI for customer service, and their top concern was that reaching a person would get harder. Gartner advises that a chatbot that cannot solve the problem should hand over to an agent chat that “picks up where the chatbot left off”. That is a journey requirement.
The US Consumer Financial Protection Bureau’s report on chatbots in consumer finance names the same pattern “doom loops”, often caused when an issue falls outside the chatbot’s limited capabilities.
Five failure patterns cause most of the damage:
| Failure pattern | What the customer experiences | Why a conversation score misses it | Journey signal to watch |
|---|---|---|---|
| “Talk to a human” ignored | Asks for a person, gets more bot answers, or a dead end | The bot stayed polite and on-topic, so its conversation passes | Explicit human request with no human contact in the journey |
| Channel loop | Chat says call, phone says use chat, nobody solves it | Each contact “correctly” routed the customer somewhere | Same issue, 2 or more channel switches, no resolution |
| Context lost at a handoff | Repeats the order number, the problem, the steps already tried | The human agent was friendly and asked the right questions | Customer gives the same detail twice in one journey |
| Repeat contact | The issue was “resolved”, then comes back days later | The first conversation ended with a confident answer | New contact on the same issue within the window |
| Silent abandonment | Gives up mid-journey, maybe cancels or churns | No complaint, no survey, the ticket closes as solved | Journey ends with no resolution and no further contact |
Customers in public support forums describe the channel loop in a sentence: chat sends them to the phone line, and the phone line sends them back or hangs up. Neither the chat transcript nor the call, scored alone, would fail. For the failures inside a single AI conversation that dashboards count as wins, see our guide to silent AI agent failures.
How do you define a customer journey for QA?
Define a customer journey as every contact from the same customer about the same issue within a fixed window, for example 7 days from the first contact. You need three rules written down before you review anything: how you recognize the same customer, how you recognize the same issue, and when a journey counts as ended.
Same customer. Use the strongest identifier you have: customer ID or account number, then verified email, then phone number. Count anonymous chats separately rather than guessing.
Same issue. Match on the intent tag, the order or case number, or a linked ticket. When tags are unreliable, have a reviewer confirm matches on a sample first.
The window. Pick it by issue type, then keep it fixed:
| Issue type | Suggested window | Why |
|---|---|---|
| Simple how-to or account questions | 3 days | If it comes back later, it is usually a new question |
| Orders, deliveries, returns | 7 days | Covers the wait for a delivery or a refund to show |
| Billing, refunds, disputes | 14 days | A wrong answer shows up on the next statement or payment |
| Claims, cancellations, regulated cases | 30 days | Long processes with several required steps |
These windows are our starting rules, not an industry standard. Plot the days between first and second contact on the same issue, and set the window where most real follow-ups have arrived.
How a journey ends. Resolved (the actual problem was solved and nothing followed within the window), abandoned (the customer stopped without a resolution) or open (review it later). First contact resolution uses the same repeat window but only asks about the first contact; journey QA judges the whole path.
Which journey-level metrics should you track?
Track seven journey metrics: the share of journeys resolved correctly, contacts per journey, handoff failure rate, context-lost rate, human requests honored, silent abandonment rate and journey time to resolution. Together they show whether customers get solved, how much effort it costs them, and which seam breaks most often. Report them per issue type and per entry channel.
| Metric | How to calculate it | Starting rule (ours, tune to your baseline) |
|---|---|---|
| Journeys resolved correctly | Journeys where the actual problem was solved correctly / closed journeys | Investigate any issue type below 85% |
| Contacts per journey | All contacts, every channel / journeys | Investigate any issue type above 1.5 |
| Handoff failure rate | Handoffs that lost context, came too late or sent the customer back / reviewed handoffs | Aim for under 5% |
| Context-lost rate | Multi-contact journeys where the customer repeated a detail / multi-contact journeys | Investigate above 5% |
| Human requests honored | Explicit requests for a person followed by a human contact / all such requests | Aim for 95% or more |
| Silent abandonment rate | Journeys ended unresolved with no complaint / closed journeys | Review a sample weekly, whatever the rate |
| Journey time to resolution | First contact to resolution, median and 90th percentile | Watch the 90th percentile |
A high containment rate can hide a poor journey: a customer who leaves the bot and calls the phone line still counts as contained. Contacts per journey catches that. And keep “resolved” apart from “resolved correctly”: a confident wrong answer looks resolved until the repeat contact arrives. Our AI agent resolution audit shows how to check.
The journey QA scorecard (copy this)
The journey QA scorecard is a short set of criteria that only make sense across a whole journey. Each criterion has a clear fail condition and names the data a reviewer needs. Score each conversation on your normal scorecard as usual; score the journey on these criteria on top, so one never replaces the other.
| # | Journey criterion | What counts as a fail | Data you need |
|---|---|---|---|
| 1 | The actual problem was solved correctly by the end | Problem unsolved, solved wrongly, or “solved” and back within the window | All contacts in the window, the final outcome, the policy or knowledge article that applies |
| 2 | Requests for a human were honored (critical) | Customer asked for a person and did not get one, or got one only after asking 2 or more times | Bot and AI agent transcripts, escalation log |
| 3 | No loop between bots, channels or teams (critical) | The customer was sent back to a channel or agent they had already tried for the same issue | Channel and handler per contact, routing log |
| 4 | Context carried across every handoff | The customer repeated an identifier, the problem or steps already tried | Transcripts on both sides of each handoff, handoff summary if any |
| 5 | Escalation happened on the right trigger | Escalated too late (after repeated failed answers) or bounced to a human with no reason given | Escalation rules, transcript at the point of handoff |
| 6 | Consistent answers across channels and handlers | Two contacts gave the customer different answers on the same policy | All answers given, the policy that applies |
| 7 | Promises were kept | A callback, refund or follow-up promised in one contact did not happen | Promises in transcripts, order or case system |
| 8 | Effort was proportionate | More contacts or days than the issue type needs (above your contacts-per-journey or time rule) | Contact count, timestamps |
| 9 | Ending was explicit | The journey ended in silence with the issue unsolved (silent abandonment) | Last contact, outcome, any churn or cancellation event |
Scoring rule. A journey fails if it fails criterion 1, 2 or 3 (the critical ones). Otherwise, count the remaining fails: 0 is a pass, 1 is a pass with a finding, 2 or more is a fail. Record one root-cause segment for every fail: the bot, a named AI agent, the routing or handoff rule, a channel, a human agent, or the policy or knowledge content. That attribution is what turns a journey score into a fix.
Criterion 4 is the same continuity check our guide to scoring the human-AI handoff covers in depth, one seam at a time. For journeys where several AI agents pass a case between them, our explainer on AI agent orchestration covers the routing patterns behind criteria 3 and 5.
How do you sample and review whole journeys?
Sample journeys in two layers: review every journey that trips a risk signal (an ignored human request, 3 or more contacts, 2 or more channels, a loop, or an abandonment), then add a random sample of ordinary journeys to measure the true failure rate. Review each journey in time order, on one timeline, with every contact side by side.
Layer 1: risk-triggered journeys. Failures concentrate here, so read all of them, or a fixed number per trigger per week:
- An explicit request for a person (“agent”, “human”, “representative”) in a bot or AI agent transcript.
- 3 or more contacts, or 2 or more channel switches, on the same issue.
- A repeat contact after a contact marked resolved.
- A journey that ends unresolved with no further contact.
Layer 2: a random sample. Triggers show where to look, not how often things go wrong. To report a failure rate you can defend, add a random sample of closed journeys; 150 to 400 a month gives a usable margin of error for one issue type. Our piece on QA sampling statistics explains why smaller samples mislead.
How to review one journey. Rebuild the timeline: every contact in date order, with channel and handler (bot, AI agent, human, phone menu). Read the first and last contacts before the middle; the gap between what they asked for and what they ended with is the verdict. Then walk each seam. Calls need a transcript or summary to sit on the same timeline; many contact centers already score a sample of calls, and the journey review places them next to the chat and bot contacts.
The journey review template
Copy this into a spreadsheet or review form. One block per journey.
| Field | What to fill in |
|---|---|
| Journey ID | Your own ID, or the first ticket number |
| Customer and issue | Customer identifier, issue type, order or case number |
| Window | First contact date, window length, end state (resolved, abandoned, open) |
| Timeline | One row per contact: date and time, channel, handled by (bot, named AI agent, human, phone menu), what happened in one line, context carried in (yes or no), outcome |
| Risk triggers | Which Layer 1 triggers fired, if any |
| Journey scorecard | Pass or fail on criteria 1 to 9, with the contact number as evidence |
| Verdict | Pass, pass with finding, fail |
| Root-cause segment | Bot, AI agent (name), routing or handoff rule, channel, human agent, policy or knowledge |
| Owner and fix | Who owns that segment, the fix agreed, the date it ships |
| Re-check | The date you will review the same journey type again |
A worked example (illustration, numbers invented to show the math)
A support team handles 50,000 contacts a month across a chatbot, an AI agent in chat and email, a phone line and human agents. Joined into journeys with a 7-day window, those contacts form 38,000 journeys:
- 31,800 journeys (84%) take one contact.
- 6,200 journeys take two or more, with 18,200 contacts between them (2.94 each).
- Contacts per journey overall: 50,000 / 38,000 = 1.32.
Conversation QA looks healthy: 91% of sampled conversations pass. Then the QA lead reviews a random sample of 150 multi-contact journeys. 48 fail (32%):
| Root cause | Failed journeys | Share of fails |
|---|---|---|
| Context lost at the bot-to-human handoff | 17 | 35% |
| Request for a human ignored or delayed | 12 | 25% |
| Loop between chat and phone | 9 | 19% |
| Wrong resolution, customer came back | 6 | 13% |
| Different policy answers in chat and on the phone | 4 | 8% |
| Total | 48 | 100% |
Inside those 48 failed journeys sit 141 conversations, and 118 of them (84%) passed the conversation scorecard. The journeys failed while most of the conversations in them passed.
Scaled up, 32% of 6,200 is about 1,980 failed journeys a month. The 95% margin of error is about plus or minus 7.5 points (1.96 × √(0.32 × 0.68 / 150) = 0.075), so the true number sits between roughly 1,520 and 2,450. Either end is worth fixing. And 29 of the 48 fails (60%) come from two seams, the handoff payload and the human-request rule, so that is where the first fixes go.
See it on your own conversations. Kaizo scores 100% of your support conversations, human and AI agent, against your own scorecard, and links every score to the evidence in the conversation. Book a demo.
Who should own customer journey quality?
One person should own journey quality end to end, usually the Head of Support or Support Operations, because no channel or agent owner sees the whole path. That owner holds the journey metrics and the weekly review. Each failed journey then goes to the owner of the segment that caused it.
| Role | Owns | Acts on |
|---|---|---|
| Journey owner (Head of Support or Support Ops) | The journey metrics, the journey scorecard, the weekly review | Priorities across teams; escalates when a fix stalls |
| AI agent owner (per bot or AI agent) | The agent’s instructions, knowledge and escalation triggers | Fails attributed to that agent: wrong answers, ignored human requests |
| Routing and handoff owner | Routing rules, handoff payload, transfer between channels | Loops, lost context, late escalation |
| Channel leads (chat, email, phone) | Each channel’s process and staffing | Channel loops, broken callbacks, inconsistent answers |
| QA lead | Sampling, reviewer calibration, the review template | Reliable journey scores; new failure patterns |
| Knowledge and policy owner | The content humans and AI agents answer from | Inconsistent answers across channels |
Two rules keep this from becoming a blame meeting. Every failed journey gets exactly one root-cause segment; record the rest as secondary findings. And the fix depends on who failed: a person gets coaching, an AI agent gets a change to its instructions, knowledge or triggers. See coaching humans vs fixing AI agents.
How do you fix the failures journey QA finds?
Fix journey failures at the seam that caused them, not in the conversation where they surfaced. Most fixes are rule changes: an escalation trigger, a handoff payload, a policy that says when a request for a human is always honored, or one shared answer for a policy across channels. Then re-review the same journey type to confirm the fix held.
| Failure found | Typical fix | Where to read more |
|---|---|---|
| “Talk to a human” ignored | A hard rule: an explicit request, or a second failed answer, hands over with a summary | AI agent guardrails |
| Channel loop | Remove “please call us” or “please use chat” answers for issues the other channel cannot solve either; offer a callback or a ticket instead | Why AI conversations fail |
| Context lost at handoff | A fixed handoff payload: customer ID, issue, steps tried, what the customer wants | Scoring the human-AI handoff |
| Inconsistent answers | One policy source that both people and AI agents answer from, versioned | AI agent policy management |
| Wrong resolution, repeat contact | Tighten what the AI agent may resolve alone; audit its resolutions | AI agent autonomy levels |
| Silent abandonment | Review abandoned journeys weekly; add a follow-up for unresolved high-risk issues | AI agent monitoring |
Test each change before it reaches customers, then measure it in production. Our guide to AI agent evaluation covers both: test sets before launch, and scoring live conversations after.
When is journey QA not for you?
Journey QA is not worth the setup when most issues are solved in one contact and you have no bot or AI agent in front. If contacts per journey sit near 1.1, customers rarely switch channels, and you have a few hundred contacts a week, conversation QA plus a simple first contact resolution number already shows what you need.
It is premature when you cannot join contacts to a customer: if most chats are anonymous and calls carry no customer identifier, fix that first. Meanwhile, read every transcript with an explicit request for a human, which needs no joining. And journey QA never replaces conversation QA: one shows where the path broke, the other what went wrong inside a contact.
How to start this week
You can start journey QA this week with a spreadsheet and one issue type. The goal of week one is not a dashboard. It is finding the one seam that breaks most journeys, so the first fix pays for the effort.
- Pick one high-volume issue type where a bot or AI agent takes the first contact, such as order status or refunds.
- Write the three rules: which identifier, which issue tag or case number, which window.
- Pull 30 journeys: 20 that trip a risk trigger and 10 at random.
- Review them on the journey scorecard and template, with one root-cause segment per fail.
- Name the owner of the worst seam and agree one fix with a date.
- Repeat next week on the same issue type to see whether the fix held.
When the manual review proves the pattern, automate what scales. Kaizo’s AI agent QA scores AI agent and human conversations on the same scorecard, including whether the agent escalated on the right trigger and passed on the context, and AutoPilot scores every conversation the moment it resolves in Zendesk or Salesforce. Joining those conversations into journeys comes from your helpdesk data: linked tickets, customer IDs and issue tags. For the bigger picture, see our guide to agentic customer service.
Frequently asked questions
What is a good number of contacts per journey?
There is no universal benchmark, because it depends on the issue type. Use your own baseline per issue type and investigate anything that rises. As a starting rule, simple questions should sit close to 1.0 and anything above 1.5 deserves a look at the journeys behind it.
Is customer journey QA the same as customer journey mapping?
No. Journey mapping designs the path you want customers to take, often for marketing or service design. Journey QA checks the paths customers actually took, contact by contact, and judges whether each one solved the problem without loops, lost context or ignored requests for a person.
Can you review voice and chat journeys together?
Yes, if each call has a transcript or a written summary in the same place as your chat and email contacts. The journey review then puts the call on the same timeline as the bot and chat contacts and applies the same journey criteria. Without a transcript or summary, a reviewer can only judge the call from its outcome.
Is journey QA the same as AI agent monitoring?
No. Monitoring watches live signals for one AI agent, such as escalation rate, errors and drift, and alerts when they move. Journey QA reviews finished journeys across every bot, agent, channel and person, and judges whether the customer’s problem was solved. Monitoring tells you something changed; journey QA tells you what it cost the customer.
How do you detect silent abandonment?
Look for journeys that ended unresolved with no further contact: a bot conversation that stopped mid-flow, an escalation the customer never answered, a callback that never connected. Some of these customers solved it elsewhere; some cancelled. Review a weekly sample and, where you can, check cancellations or churn in the weeks after.
