When people and AI agents share the support queue, hold both to one quality standard, then fix them differently. Humans improve through coaching, practice and feedback. AI agents improve through knowledge fixes, policy and instruction changes, tighter tool permissions, and a regression test that proves the fix. Same scorecard, two repair paths, one weekly loop.
In short
- Score humans and AI agents on the same core criteria (outcome, accuracy, policy, tone, handoff), so their results can be compared and the customer gets one standard.
- Add a small block of checks per handler: grounding and scope for the AI agent, case notes and follow-up for people.
- A human miss is usually one person on one day; an AI agent miss repeats in every conversation that hits the same setup, so the fix goes into the setup, not into a conversation.
- Compare the two only intent by intent and on the same intent mix, or the AI agent looks better simply because it gets the easy work.
- Every fix has a named owner and a check: the person’s next scored conversations, or the AI agent’s regression set and next week’s live scores.
Building the shared scorecard is covered in one scorecard for human and AI agents, and running a coaching session in what agent coaching is. This guide covers what happens after a miss shows up.
Why should humans and AI agents share one quality standard?
Humans and AI agents should share one quality standard because the customer experiences one service: a wrong refund amount hurts the same whether a person or a bot gave it. A shared standard also lets you decide which intents the AI agent keeps on evidence instead of containment rates.
Without it, the AI agent gets a softer test (containment counts a win when the customer did not reach a human, not when the answer was right), people get blamed for late, context-free handovers, and nobody can decide whether the bot should keep warranty claims.
Helpdesks now ship this as a feature. Kustomer describes one weighted scorecard where “that same scorecard scores every conversation, AI agent and human rep alike” (Kustomer). The scorecard is the easy part; the hard part is what you do with a failed criterion.
Which scorecard criteria apply to both, and which to only one?
Outcome, accuracy, policy, identity checks, tone and handoff apply to both, with the same wording and weight. Grounding in your content, staying in scope and admitting uncertainty belong in an AI agent block. Case notes and kept promises belong in a human block. Score the shared core for comparison and report the blocks next to it.
| Criterion | Humans | AI agent | Why |
|---|---|---|---|
| Issue resolved or clear next step | Yes | Yes | The customer outcome is the same job |
| Every fact correct | Yes | Yes | A wrong fact costs the same either way |
| Policy followed (refunds, credits, exceptions) | Yes | Yes | Policy does not change with the handler |
| Identity verified before account data | Yes | Yes | Same risk, same rule |
| Tone fits the customer and the brand | Yes | Yes | Same brand voice |
| Handoff passes context | Yes | Yes | Both hand over, both receive |
| Answer grounded in your published content | No | Yes | People use judgment; a model can invent |
| Stays within allowed intents and actions | No | Yes | Scope is a setting for the bot |
| Says “I don’t know” instead of guessing | Rarely | Yes | Guessing is the AI agent’s typical failure |
| Case notes let the next person continue | Yes | No | The AI agent’s handoff payload covers this |
| Promised follow-up happened | Yes | No | Usually only people promise callbacks |
Some human criteria make no sense for a bot (schedule adherence, after-contact work); our AI agent scorecard lists what to drop and add. The rule that matters: a shared criterion gets one definition. If “policy followed” means something looser for the AI agent, the comparison is gone.
Why does the same miss need two different fixes?
Because the cause sits in different places. A person who gets a refund rule wrong has a gap in knowledge, skill or habit, and coaching closes it. An AI agent that gets it wrong runs a setup that produces the error (content, instructions, tools or routing), and only changing the setup fixes it.
| Human agent | AI agent | |
|---|---|---|
| How a miss spreads | One person, some of the time | Every conversation that hits the same setup |
| Where the cause lives | Knowledge, skill, judgment, workload | Knowledge source, instructions, policy rules, tool permissions, routing |
| How it improves | Feedback, practice, an example to copy, time | A change to the setup, tested before release |
| How a fix can backfire | Rarely; a coached skill stays | Often; a new instruction can break three intents that used to pass |
The last row is why AI agent fixes need a regression test: a changed instruction or article can fix one case and break cases that used to pass. Anthropic’s guidance on agent evals says regression evals “should have a nearly 100% pass rate”, and that to start an eval set, “20-50 simple tasks drawn from real failures is a great start” (Anthropic). For a support team, those tasks come from your own scored conversations.
People, in contrast, learn from what works for others. A study of 5,179 customer support agents found an AI assistant raised issues resolved per hour by 14% on average and by 34% for novice and low-skilled workers, with suggestive evidence that it spread the practices of the strongest agents (Brynjolfsson, Li and Raymond, NBER). Coaching with a colleague’s best conversation uses the same mechanism.
One shared cause is easy to miss: humans and AI agents often read the same help center and policy pages. When both fail the same intent, the fix is usually one article or one policy. Our guide on why AI conversations fail traces those root causes.
The same miss, two fixes (copy this table)
This table maps a failed criterion to the fix for a person, the fix for an AI agent, and how to prove each worked. Use it as the triage sheet in your weekly review; the verify column turns a fix into a closed item rather than a hope.
| Miss seen on the scorecard | Human fix | AI agent fix | How to verify |
|---|---|---|---|
| Wrong fact (refund window quoted as 60 days, policy says 30) | Show the policy and the conversation; two practice cases on the same intent; fix the macro if it is wrong | Correct or retire the outdated article the agent read; remove conflicting versions | Human: accuracy on the next 20 scored conversations of that intent. AI: the failed conversation becomes a test case, the full set passes, live accuracy on the intent next week |
| Policy breach (goodwill credit above the approved limit) | Coaching on the limit and the approval route; a manager sign-off step for exceptions | Cap the amount in the refund tool itself, not only in the instructions; restate the policy rule | Human: no breach in 4 weeks. AI: a test case that pushes for a bigger credit fails safely; tool logs show nothing above the cap |
| Missed or late escalation (kept a legal threat in the bot) | Role-play the escalation triggers; add them to the checklist | Add the trigger to the escalation rules; route the intent to people if the trigger fires | Human: escalation criterion passes on flagged cases. AI: test cases with each trigger phrase hand over on the first mention |
| Handoff without context (customer repeats the story) | Case note template; review one handoff together | Add order number, intent and steps tried to the handoff summary | Receiving agent does not ask again, scored on the next 20 handoffs |
| Cold tone with an upset customer | Listen to a peer’s strong example; practise acknowledging first | Add tone instructions and two example replies for negative sentiment | Tone pass rate on conversations with negative sentiment, before and after |
| Closed but not resolved | Confirm-before-close habit; review the reopened cases | Define what “resolved” means per intent in the instructions; stop ending on a generic sign-off | Reopen rate within 7 days on that intent, plus the resolution criterion |
| Account data shared before identity check (auto-fail) | Same-day one-to-one; compliance refresher | Make the account lookup tool require a verified flag, enforced in the integration, not the prompt | Human: zero repeats. AI: adversarial test cases cannot reach account data; zero live occurrences |
| Guessed an answer it did not know | Make “I’ll check and come back” acceptable; show where to look | Instruct and test the fallback reply; log the question as a knowledge gap | Test questions with no answer in the knowledge base get the fallback; the gap is filled |
Two patterns stand out. The AI agent fixes lean on hard limits (a tool cap, a required flag) wherever a miss is costly, as OWASP’s guidance on excessive agency advises: minimal functions and permissions, and “require a human to approve high-impact actions before they are taken” (OWASP); our AI agent guardrails guide covers which limits belong in code. And the human fixes are small: one intent, one example, one practice case. A session that covers ten criteria at once changes none of them.
See both on one scorecard. Kaizo scores AI agent conversations on the same scorecard as your team and links every score to the lines behind it, so each miss is an example someone can fix. Book a demo.
How do you compare human and AI agent results fairly?
Compare human and AI agent results intent by intent on the shared core, then weight both by the same intent mix. A raw average mostly shows who got the easy work. Only compare intents where both handled enough conversations, and attribute a failed handoff to the side where the miss started.
Here is an illustration with numbers we made up to show the arithmetic, not customer data. The criterion is “policy followed”, as a pass rate:
| Intent | AI agent conversations | AI agent pass rate | Human conversations | Human pass rate |
|---|---|---|---|---|
| Returns | 4,000 | 97% | 500 | 95% |
| Warranty claims | 500 | 80% | 2,000 | 92% |
| Raw total | 4,500 | 95.1% | 2,500 | 92.6% |
The raw totals say the AI agent is 2.5 points better. Now weight both by the combined mix, 4,500 returns and 2,500 warranty claims out of 7,000:
| Mix-adjusted pass rate | Calculation | Result |
|---|---|---|
| AI agent | (4,500 x 97 + 2,500 x 80) / 7,000 | 90.9% |
| Human team | (4,500 x 95 + 2,500 x 92) / 7,000 | 93.9% |
On the same mix, the human team is 3.0 points better. The useful reading is per intent, and it points to a fix, not a verdict: the AI agent is fine on returns but fails one in five warranty claims, so warranty goes into the weekly loop. Is the article clear, does the agent have the purchase date, should the intent route to people until it passes?
Three more rules keep the comparison honest:
- Minimum volume. Below about 50 scored conversations on one side in the period, report “not enough data” (our suggested starting point, not a statistical law).
- Handoff attribution. When a conversation moves from the AI agent to a person, tag the side where the first failed criterion happened, or people absorb the bot’s misses. Customer journey QA covers scoring a case across handlers.
- Same scorer. Score both sides the same way. A human-reviewed team against an auto-scored bot is two standards again.
Who owns each fix?
Each fix needs one named owner: team leads for human skill gaps, the AI agent owner (often a conversation designer or CX operations lead) for instructions and tool settings, the knowledge manager for articles both sides read, the policy owner for the rule itself, and the QA lead for the scorecard.
| Cause found | Owner | Needs approval from | Applies to |
|---|---|---|---|
| One person’s skill or habit | Team lead | None | Human |
| The same miss across several people | Team lead plus process or knowledge owner | Support manager | Human (and often AI) |
| Wrong, missing or conflicting article | Knowledge manager | Policy owner if the rule changes | Both |
| Unclear or contradictory policy | Policy owner (finance, legal, CX) | Head of CX | Both |
| AI agent instructions or prompt | AI agent owner | QA lead signs off the regression run | AI agent |
| Tool permissions and limits | AI agent owner | IT or security | AI agent |
| Escalation and routing rules | CX operations | Head of CX | Both |
| A criterion that scores the wrong thing | QA lead | Calibration group | Both |
The approval column matters most for AI agent changes: a prompt edit reaches every conversation the next minute, so it gets the same release discipline as a policy change (AI agent policy management covers who signs off which rule). NIST’s AI Risk Management Framework is built on four functions, govern, map, measure and manage (NIST); this table is a support team’s version of govern.
What does the weekly improvement loop look like?
One rhythm for both sides: pull the week’s misses by criterion and intent, decide whether each is a person, a process or the AI agent’s setup, assign the owner, fix, and check last week’s fixes. Humans get coaching within days; AI agent changes ship only after their regression set passes.
| Day | Step | Human side | AI agent side |
|---|---|---|---|
| Monday | Pull the misses | Failed criteria per person and intent | Failed criteria per intent, with the conversations |
| Monday | Triage | One person or many people? | Setup, knowledge or policy? |
| Tuesday | Assign | Coaching card per person, one or two criteria each | One ticket per pattern, with the failed conversations attached |
| Tuesday to Wednesday | Fix | 15-minute one-to-ones, practice cases, peer examples | Change made; failed conversations added as test cases; full regression set run |
| Thursday | Release | Not needed | Release only if the gate passes; log the change and the date |
| Next Monday | Verify | Same criterion, same person, last 7 days | Same criterion, same intent, last 7 days |
The triage step needs a decision rule. Ours:
- One person misses it, others on the same intent pass: coach the person.
- Three or more people miss the same criterion on the same intent: stop coaching and look at the article, the macro or the process. Coaching ten people around a broken article is the most expensive fix there is.
- The AI agent misses it and people pass: fix the AI agent’s setup.
- Both miss it on the same intent: fix the shared source (the article or the policy) once, and check both sides next week.
AI-drafted coaching saves hunting for examples if a manager reviews each card (AI coaching for support agents has the checklist); the AI side’s test discipline is in AI agent testing.
How do you prove a fix worked for a person and for an AI agent?
Prove a human fix by scoring the same criterion on that person’s next conversations of the same intent, over 2 to 4 weeks. Prove an AI agent fix twice: before release, the failed conversations and the full regression set pass; after release, the same criterion on the same intent passes in live scoring the following week.
| Human fix | AI agent fix | |
|---|---|---|
| Before the fix counts | Nothing; coaching cannot be pre-tested | Regression set passes, with critical cases run 3 times and passing every time |
| After | Same criterion, same intent, next 20 or more scored conversations | Same criterion, same intent, next week of live conversations |
| Expect | Gradual change over 2 to 4 weeks | A step change, or nothing |
| If it did not work | A second session with a different example, then look at workload or the process | Reopen the ticket; the cause was probably somewhere else in the setup |
Critical cases run several times because a model can pass a case once and fail it on the next run. Anthropic calls the “all runs pass” measure pass^k and notes that it falls as the number of runs rises (Anthropic); for an identity check, “passes every time” is the only acceptable bar. These thresholds are suggested starting points; set your own before the first review. Live scoring of every conversation closes the loop, since a 2% sample may give you three data points per intent a week. AI agent evaluation covers how pre-release evals and live scoring fit together.
Will AI agents replace human support agents?
AI agents are taking over high-volume, well-documented intents, and human work moves toward exceptions, judgment calls, upset customers and handovers. New roles grow around the AI agent (an owner for its setup, a knowledge manager, QA for both sides), and the quality standard stays the same.
That shift changes coaching. When routine returns go to the AI agent, people handle harder conversations on average, and their scores can drop with no change in skill. Report human scores per intent, or you will coach people for a mix change.
When this approach is not for you
This loop assumes weekly volume, a scorecard you trust, and an AI agent whose content, instructions and tools you can change. It is more than you need if your AI agent handles a few dozen conversations a week (read them all and fix what you see), if you have no scorecard yet (build the shared core first), or if your team is under five people.
If the AI agent is a closed box, its side of the loop becomes a ticket to your vendor: keep the scoring and put the evidence in that ticket.
How to start this week
Start small: two intents, one shared core, last month’s conversations and one review slot next Monday. That is enough to run the triage rule on real misses and close your first fixes on both sides within a week.
- Pick two intents that both humans and the AI agent handle, one easy and one hard.
- Agree the shared core for those intents: outcome, accuracy, policy, identity, tone, handoff. One definition each.
- Score last month’s conversations on both sides and fill the mix-adjusted table above.
- Run the triage rule on the three biggest misses. Write the owner next to each.
- Use the same miss, two fixes table for each miss, including the verify column.
- Book a 30-minute review for next Monday to check this week’s fixes against the same criterion.
Kaizo is one way to run the scoring side of this loop. AutoPilot scores each conversation against your own scorecard when it resolves, with a comment that explains the score; in Zendesk, that includes the conversations Zendesk’s AI agents handle, on the same scorecard as your team. For people, Kaizo’s coaching drafts each agent’s coaching card from their ratings, linked to the conversations behind it, and a manager approves it before the agent sees it. If you want to see that on your own conversations, QA for AI agents shows how the AI agent side works.
Frequently asked questions
What is human-AI collaboration in customer service?
It is a setup where AI agents and people share the work: AI agents usually take high-volume, well-documented intents, and people take exceptions, emotional cases and handovers. It works when both are held to one quality standard and the handover carries the context.
Can you coach an AI agent?
Not in the human sense: an AI agent does not learn from a feedback conversation. It improves when someone changes its knowledge source, instructions, policy rules, tool permissions or routing, and tests that the change fixed the case without breaking others.
Do AI agents learn from their own mistakes in production?
Most support AI agents do not change their behavior from one conversation to the next on their own, so a mistake repeats until someone changes the setup. Turn the failed conversation into a test case, so the fix can be proven and the error cannot quietly return.
Should AI agents and human agents be scored on the same scorecard?
Yes, on a shared core with the same wording and weight, plus a small block of checks per handler for failures only one side has, such as invented answers for the AI agent.
How many AI agent conversations should a human review?
If an automated scorer covers every conversation, people review the failures, every auto-fail and a sample of passes to check the scorer. Without one, review enough per intent to see weekly patterns; a few percent across all intents usually misses repeat errors on the smaller ones.
