Skip to content

Best practice

Chatbot to Human Handoff: Metrics and Scorecard

How to measure chatbot to human handoff quality: the metrics with definitions, a scorecard for the bot's decision and the human's pickup, and design rules.

· Updated · 15 min read

Part of: How to QA AI Agents and Chatbots

Chatbot to human handoff QA scores the moment a conversation passes between an AI agent or chatbot and a human, in either direction. It is where blended support breaks most often, and per-agent scores miss it because the failure belongs to the transition. Measure it with a few handoff metrics and a scorecard that judges both sides.

In short

  • AI-to-human and human-to-AI handoffs fail in different ways.
  • AI-to-human fails on late escalation, lost context, an ignored request for a person, or no usable reason.
  • Human-to-AI fails when an unresolved case goes back to automation.
  • Containment and handle time cannot see it. Measure handoff rate by intent, late handoffs, context carried, repeat questions and resolution after handoff.
  • Score both sides of the seam: the bot’s decision to hand off and the human’s pickup.

Why the handoff is its own failure point

When you QA a human agent and you QA an AI agent, you are scoring two halves of a conversation that increasingly is not two halves. In a blended model a customer might start with an AI agent, get escalated to a person, and be routed back to automation for a follow-up, all in one thread. Each participant can score well on their own stretch while the conversation as a whole is a bad experience, because the damage happened at the transition and belongs to neither of them.

That is the blind spot. A per-agent scorecard evaluates what each party did with the conversation while they held it. It has nothing to say about what happened at the moment they passed it on, which is exactly where blended support tends to break. The handoff needs to be scored as a thing in itself, and it is one step of the wider practice of customer journey QA, which scores the whole path a customer takes across bots and people.

What goes wrong in a chatbot to human handoff?

Most bad chatbot to human handoffs come down to a few failures: the bot escalates too late, context is lost so the customer repeats themselves, a request for a human is ignored, the customer loops between bot and queue, the transfer is cold, or the promised person never arrives.

Regulators have noticed. The US Consumer Financial Protection Bureau’s report on chatbots in consumer finance (June 2023) describes “doom loops” when an issue falls outside what the chatbot can do, quotes one complaint (“I am being sent in an endless loop with no way out”) and warns that chatbots which prevent access to live, human support can lead to law violations.

FailureWhat the customer experiencesWhat it looks like in the transcriptMetric that catches it
Escalating too lateSeveral rounds of the same unhelpful answer before a person appearsThe bot repeats or rephrases the same answer 3 or more times, or the customer asks twice for helpLate handoff rate
Lost context“Can I have your order number?” from the human, after giving it to the botThe human asks for something already in the threadContext-carried rate, repeat-question rate
“Talk to a human” ignoredThe bot answers a different question, or offers the FAQ againAn explicit request for a person with no handoff on that turnIgnored human request rate
Handoff loopsSent to a queue, then back to the bot, then to a queue againMore than one handoff for the same issue, or the bot re-takes an unresolved caseHandoff loop rate
Cold transferSilence, then a new name, with no idea why or how long to waitNo message telling the customer a person is taking over, why, and whenCold transfer rate, time to human
Dead end“Connecting you to an agent” outside opening hours, then nothingA handoff with no human reply and no fallback offeredAbandonment during handoff

The dead end is the quiet one. Zendesk’s guide to escalation flows for AI agents puts an availability check before a messaging escalation, with email as the fallback outside working hours. Without that check, the handoff promises a person who never arrives.

AI-to-human: what to score when the agent escalates

When an AI agent escalates to a person, score three things: timing (did it hand off when it stopped making progress), context (did the human get the history and diagnosis) and reason (can the human act on it). Each failure mode deserves its own criterion.

Timing: did it escalate at the right moment

Too late is the common one. An agent that keeps trying after it is clearly stuck runs the customer in circles before handing over a frustrated person. Too early wastes the automation and the human’s time. The criterion is whether the escalation happened at the point the agent stopped making progress, which you can see in the transcript.

Context: did the human get what they need

The worst handoff makes the customer start over. Score whether the escalation carried the history, the diagnosis so far, and what the customer actually wants, so the human opens the case informed rather than cold. A handoff that drops context turns one conversation into two.

Reason: was the escalation intelligible

An escalation tagged only escalated to human tells the human nothing. Score whether the reason for the handoff was captured in a form the receiving person can act on. This is one of the criteria on an AI agent scorecard, but it only becomes visible when you read across the seam.

How should you design a chatbot to human handoff?

Design the handoff around eight rules: honor a request for a person on the first ask, escalate on no progress, send red-list topics straight to a human, check availability, pass a context package, tell the customer what happens next, never bounce an unresolved case back, and record the reason.

  1. Honor “talk to a human” on the first ask. Intercom’s escalation guidance documentation lists three default triggers for Fin: the customer clearly asks for a human, strong frustration or anger, and a customer stuck in a repetitive loop. Your own guidance and rules can override that default behavior, so check that yours did not switch one off.
  2. Escalate on no progress. Write a rule you can check in a transcript, such as the same answer given twice.
  3. Keep a red list. Some intents go to a person every time: disputes, legal complaints, vulnerable customers, anything outside the actions the bot may take alone (an autonomy level decision).
  4. Check availability. Outside opening hours, offer a callback, email or ticket and say when a human will reply.
  5. Pass a context package. At minimum: a short summary, the customer’s goal, the steps already tried, verified identity, and the reason for the handoff. Microsoft’s Copilot Studio documentation on handing off to a live agent says the agent can share the full conversation history and all relevant variables, plus an optional private message to the live agent. Your QA checks it arrived and was useful.
  6. Make it warm. Tell the customer a person is taking over, why, and roughly how long the wait is.
  7. No bounce-back. The bot does not re-take a conversation a human has not resolved; a second handoff on the same issue goes to a senior queue.
  8. Log the reason as a field. “Requested human”, “out of scope”, “policy”, “frustration”, “failed action”. Without it you cannot compute handoff rate by reason.

How do you measure AI-to-human handoff quality?

Measure AI-to-human handoff quality per intent: handoff rate, late handoffs, ignored human requests, context carried, repeat questions after handoff, time to human, handoff loops and resolution after handoff. Volume metrics come from your helpdesk; quality metrics need the transcript scored.

The thresholds below are starting points to tune on your own data, not industry benchmarks.

MetricDefinitionData sourceStarting alert
Handoff rate by intentHandoffs to a human ÷ AI conversations, per intentHelpdesk handoff event plus intent tagMoves more than 5 points week over week on one intent
Late handoff rateHandoffs that came after the escalation point (customer asked twice, or the bot repeated itself 3 or more times) ÷ all handoffsTranscript scoringAbove 10%
Ignored human request rateExplicit requests for a person not acted on at the first ask ÷ explicit requestsTranscript scoringAbove 2%
Context-carried rateHandoffs where the summary, goal, steps tried and verified identity reached the human ÷ handoffsHandoff payload or ticket fields, checked in the transcriptBelow 90%
Repeat-question rate after handoffHandoffs where the human asked for information already given ÷ handoffsTranscript scoringAbove 10%
Cold transfer rateHandoffs with no message telling the customer who takes over and when ÷ handoffsTranscript scoringAbove 5%
Time to humanTime from the handoff to the first human reply, as median and 90th percentileHelpdesk timestamps90th percentile above your chat service level
Abandonment during handoffCustomers who leave before the first human reply ÷ handoffsHelpdesk timestampsRising for 2 weeks
Handoff loop rateConversations with more than one handoff for the same issue, or a return to the bot while unresolved ÷ handoffsConversation log plus scoringAbove 3%
Resolution after handoffHandoffs resolved with no repeat contact on the same issue within 7 days ÷ handoffsHelpdesk plus contact historyFalling for 2 weeks

Two of these split the blame cleanly. When context did not arrive, the fix is the bot or the integration. When context arrived and the human asked anyway, the fix is coaching (see coaching humans versus fixing AI agents). For dashboards and alerts, see AI agent monitoring; for the wider metric set, AI customer service metrics.

The handoff scorecard: criteria for the bot and the human

Score a handoff on two blocks: the bot’s decision (trigger, timing, request honored, context package, customer told, availability) and the human’s pickup (read before replying, no re-ask, picked up in time, owned to resolution). Each criterion is pass or fail, with evidence a second reviewer can point at in the transcript. Copy it into your scorecard as a section that applies only to conversations with a handoff.

SideCriterionPassFail exampleData source
BotRight triggerEscalated on a rule you wrote (request, no progress, red-list intent), and never stayed with the bot when a rule was metKept a red-list dispute, or escalated a routine order status question it could answerTranscript plus handoff reason field
BotRight momentHanded off at the first point the rule was metCustomer asked for help twice and got the same article three timesTranscript
BotRequest honored“Talk to a human” acted on at the first ask“I can help with that!” followed by the FAQ againTranscript
BotContext packageSummary, goal, steps tried, verified identity and reason reached the humanTicket shows only “escalated to agent”Ticket fields or handoff payload
BotCustomer toldCustomer knew a person was taking over, why, and the expected waitSilence for 9 minutes, then “Hi, how can I help?”Transcript
BotAvailability checkedLive handoff only when someone is online, otherwise a fallback with a reply time“Connecting you now” at 2 a.m. with no reply until morningTranscript plus timestamps
HumanRead before replyingFirst reply refers to what the customer already said“Hi, what seems to be the problem?”Transcript
HumanNo re-askNo request for information already in the threadAsks for the order number the bot already verifiedTranscript
HumanPicked up in timeFirst human reply within your chat service levelCustomer waited past the target and leftHelpdesk timestamps
HumanOwned itResolved, or set a clear next step, without sending it back to the bot unresolvedClosed with “the assistant can help you with the rest”Transcript plus contact history
HumanFlagged a bad handoffTagged the handoff when the context was missing or wrongWorked around a broken summary without telling anyoneTicket tag

Two scoring rules keep it honest. Treat “request honored” and “owned it” (no bounce-back) as auto-fail criteria: a customer who asked for a person and did not get one is a failed conversation however polite the bot was. And attribute each failed criterion to its side, so the bot block feeds the AI agent owner and the human block feeds coaching. For the escalated ticket once a person holds it, see scoring an escalated ticket.

A worked example: what handoff metrics show that samples miss

Illustration: the numbers are invented to show the math. A team runs 20,000 AI agent conversations a month; 3,000 end in a handoff, a 15% handoff rate. They score every handoff with the scorecard above.

  • Late handoffs: 480 of 3,000, or 16%, mostly in refunds and delivery disputes.
  • Context carried: 2,550 of 3,000, or 85%. 450 handoffs reached the human without the summary.
  • Repeat questions: 690 handoffs, or 23%: 400 where the context never arrived, 290 where it arrived and the human asked anyway.
  • Ignored requests: 900 customers asked for a person; 90 of them, or 10%, were not handed off on the first ask.
  • Resolution after handoff: 2,340 of 3,000, or 78%, with no repeat contact within 7 days.

The split tells the team where to act. The 400 re-asks with no context are an integration fix. The 290 with context present are a coaching topic. The 90 ignored requests go to the owner of the bot’s escalation rules, this week.

Now compare a 2% random sample: 400 conversations, about 60 with a handoff (15%), roughly 10 late handoffs and about 2 ignored requests (90 of 20,000 is 0.45%; 0.45% of 400 is 1.8). Two cases read as an anecdote. Ninety, grouped by intent, read as a broken rule.

See it on your own conversations. Kaizo scores 100% of your support conversations and turns the findings into coaching. Book a demo.

Human-to-AI: the direction nobody scores

A human-to-AI handoff fails when a person passes a case back to automation before it is resolved, delegates a task the AI cannot pick up, or triggers a follow-up that contradicts what the human just did. The reverse handoff is newer, less examined and easy to miss.

Human-to-AI handoffWhat good looks likeThe failure to score for
Returning a case to automation for follow-upThe AI has the resolution and the next step is genuinely routineThe human hands back an unresolved case to clear their queue
Delegating a sub-task mid-conversationThe AI can complete the specific task and hand back cleanlyThe AI cannot pick up the thread and the customer loses continuity
Post-resolution automated follow-upThe follow-up matches what was actually resolvedThe automation contradicts or forgets what the human just did

When AI agents also hand off to each other, the same seams multiply; our guide to AI agent orchestration covers agent-to-agent handoffs.

How to score a handoff in practice

To score a handoff in practice, evaluate the whole thread across the transition, cover every handoff rather than a sample, and attribute each failure to the side that caused it.

Evaluate the whole thread

Score continuity across the seam: did context survive, did the customer avoid repeating themselves, did the second party pick up where the first left off. This is impossible if your QA looks at the AI portion and the human portion as separate records, which is how most tooling stores them.

Cover every handoff, because they are rare per pair but common overall

Any single agent pair may hand off infrequently, so a sample per agent misses handoffs almost by construction. Across the operation they are everywhere. Scoring 100% of conversations is what makes handoff failures visible as a pattern rather than as the occasional complaint. At UiPath, Kaizo automated 100% of QA with 200% ROI and an 8% lift in quality score.

Attribute each failure to one side

Was it the AI’s fault for escalating badly, or the human’s for handing back a mess? Score the bot block and the human block as separate criteria, and point every finding at the exact turns where continuity broke, so you fix the routing rule, the context package or the coaching topic. The full workflow is in how to QA AI agents and chatbots.

When handoff scoring is not for you

Skip a dedicated handoff scorecard when your bot hands off a few dozen conversations a month: read every one by hand instead. It is also premature when the bot has no written escalation rules, because “right trigger” and “right moment” need a rule to score against, or when the handoff is a link to a contact form: there is no seam until the bot and the human share a conversation.

How to start measuring handoff quality this week

Start with a manual pass: count last month’s handoffs by intent, read 30 of them end to end, tag each against the scorecard above, and split the re-asks by cause. A few hours give you a baseline before you automate anything.

  1. Pull last month’s handoffs from your helpdesk and count them by intent.
  2. Read 30 handoff conversations end to end and tag each against the scorecard above.
  3. Split the re-asks into “context missing” and “context present”. Send the first list to the bot owner and the second to your team leads.
  4. Write down your escalation rules and add a reason field to the handoff.

When the manual pass proves the pattern, automate it. Kaizo’s AI agent QA scores AI agent conversations on the same scorecard as your team’s and checks the handoff as criteria on it: did the AI agent escalate on the right trigger, pass on the context, and did the human agent pick it up. AutoPilot scores every conversation the moment it resolves. To score the whole path, see customer journey QA.

Frequently asked questions

What is handoff QA?

Handoff QA is evaluating the moment a conversation passes between an AI agent and a human, in either direction. It treats the transition as its own unit of evaluation, because a handoff can fail (lost context, late escalation, an unresolved case handed back) even when both the human and the AI score well on their own segments.

What should you score in an AI-to-human escalation?

Three things: timing (did it escalate at the point it stopped making progress, rather than too late or too early), context (did the human receive the history and diagnosis so the customer does not start over), and reason (was the escalation captured in a form the receiving person can act on).

What goes wrong in a human-to-AI handoff?

The main failures are a human handing an unresolved case back to automation to clear their queue, delegating to an AI that cannot pick up the thread and so breaks continuity, and post-resolution automated follow-ups that contradict or forget what the human just did. This direction is newer and rarely scored.

Why do per-agent QA scores miss handoff problems?

Because the failure lives between the records. A per-agent scorecard evaluates what each party did while holding the conversation, not what happened at the transition. Scoring the handoff requires reading the whole thread across the seam, which is why it needs the conversation stored and evaluated as one unit rather than two.

What is the difference between a warm and a cold handoff?

In a warm handoff the customer is told a person is taking over, why, and how long the wait is, and the human receives a summary before replying. In a cold handoff the conversation lands in a queue: the customer waits without knowing what happens, and the human starts from nothing.

What is a good chatbot to human handoff rate?

There is no single good rate, because it depends on the intent: 40% can be right for billing disputes and alarming for password resets. Track the rate per intent, watch for changes, and judge it together with late handoffs and resolution after handoff.

In KaizoKaizo Quality AssuranceKaizo scores 100% of your support conversations, human or AI, against your own quality criteria. Fully automated, or with human review where it matters.See Kaizo Quality Assurance

On this page

See this in a live demo

We will walk you through Kaizo with realistic sample conversations, so you see the example scored, explained and coached.

Trusted by global support teams

  • Foot Locker
  • Canva
  • GetYourGuide
  • Instacart
  • BlaBlaCar
  • UiPath