One scorecard for human and AI agents is a single quality standard with two layers: a shared core of criteria both are scored on, with the same definitions and weights, and a short block per handler for failures only one of them produces. You compare humans and bots on the shared core only, topic by topic, and report both on one page.
In short
- One scorecard for human and AI agents means one shared core plus a small block per handler, not an identical list for both.
- The shared core only works if each criterion has one definition, one scale and one weight for both.
- Seven rules keep it fair, from writing criteria for the transcript to deciding who owns each failed line.
- Compare on the shared core by topic: in a worked example, the raw average makes the bot look 7.9 points better while humans are 3.8 points better on a fair basis.
- Build it in six steps, starting from the scorecard your team already uses.
This page is about the shared standard. If you want the list of criteria a bot needs that a person does not, read our guide to the AI agent scorecard first. For how Kaizo applies one scorecard to your team and your AI agents, see AI agent QA.
Why use one scorecard for human and AI agents?
Use one scorecard for human and AI agents because your customers get one standard of service, whoever answers. Two separate scorecards produce two numbers nobody can compare, and two arguments about which one counts.
Three practical reasons come up in every team that runs both:
- Leadership asks one question. “Is the bot as good as our people?” can only be answered on a shared measure. With two scorecards, the answer depends on whose definitions you use.
- Routing decisions need a comparison. Deciding which topics the AI agent should handle, and which go to people, requires both scored the same way on the same topic.
- One standard is easier to keep. When a policy changes, you update one criterion, not two scorecards maintained by two teams that drift apart.
The trap is the opposite mistake: pointing your human scorecard at the bot unchanged. Greetings and spelling rarely fail for a bot, while invented policy rarely fails for a person. One standard means one core, not one identical list.
What goes in the shared core, and what does not?
The shared core holds the criteria that describe a good outcome for the customer, whoever answered. Criteria that describe how one kind of handler fails go in that handler’s block.
Here is a template you can copy. Weights are an example; the point is that the shared core has the same weight for both handlers:
| Block | Criterion | Pass when | Weight |
|---|---|---|---|
| Shared core | Correct answer | Every fact in the answer is right for this customer’s case | 25 |
| Shared core | Complete answer | All parts of the question are answered, including conditions | 15 |
| Shared core | Policy followed | Nothing is offered, promised or refused outside written policy | 20 |
| Shared core | Clear next step | The customer knows what happens next, or the issue is resolved | 15 |
| Shared core | Identity and data | Identity is verified before account details are shared | 10 |
| Shared core | Tone fits the brand | The reply reads like your brand and fits the customer’s situation | 5 |
| Shared core | Handoff | Context is passed on at a transfer, and the receiver uses it | 10 |
| AI agent block | Grounded in our content | Every fact comes from a help article or policy you publish | Pass or fail |
| AI agent block | Scope | It stays within the topics and actions it is allowed to handle | Pass or fail |
| AI agent block | Escalation trigger | It hands over on the defined triggers, and not too late | Pass or fail |
| AI agent block | Uncertainty | It says it does not know, instead of guessing | Pass or fail |
| Human block | Case notes | Notes and tags let the next person continue without asking again | Pass or fail |
| Human block | Follow-up kept | A promised callback or update happened on time | Pass or fail |
The shared core adds up to 100, which gives you one comparable score per conversation. The handler blocks are pass or fail, reported next to the score as their own pass rates. A failed handler criterion can still trigger action, such as an auto-fail for the bot on an invented refund, without distorting the comparison.
Kaizo scorecards take your own criteria, written in your words, and can be scoped per team, queue or client. That is how one shared core and two handler blocks live in one standard rather than two documents.
Seven rules that keep one scorecard fair
A shared scorecard stays fair when both handlers are scored on the same words, scale and weights, compared like with like, and charged only for their own lines.
Rule 1: One definition per shared criterion
Write each shared criterion once, in words that do not assume a person or a bot.
“The agent showed product knowledge” assumes a person. “Every fact in the answer is right for this customer’s case” works for both. Our guide to writing a QA rubric an AI can score shows how to phrase criteria so they can be checked against the transcript.
Rule 2: One scale
Score the shared core on the same scale for both, whether that is pass or fail per criterion or points per criterion. Mixing a 1 to 5 scale for people with pass or fail for the bot makes the totals look comparable when they are not.
Rule 3: One set of weights on the core
Give each shared criterion the same weight for both handlers. If you weight tone higher for people, the human total rises for reasons that have nothing to do with the customer. Our post on how to weight QA scorecard criteria covers how to set weights from outcomes, not habit.
Rule 4: Compare on the core, by topic
Compare humans and the bot on the shared core only, topic by topic. The worked example below shows why an overall average misleads.
Rule 5: Each failed line belongs to whoever wrote it
A bot’s error never lands on a human agent’s score. If the AI agent promised a refund and the human agent had to withdraw it, the failed “policy followed” belongs to the bot’s conversation, and the human is scored on how they handled the withdrawal.
Rule 6: Score the handoff as a transfer
The handoff criterion sits in the shared core because both sides can fail it: the bot by passing no context, the person by not reading it. Score each side on its own part. Our guide on scoring the human-to-AI handoff covers both directions.
Rule 7: One calibration, one change process
Calibrate your reviewers and your automated scorer on a mixed set of human and AI conversations, so “correct answer” means the same thing on both. When a shared criterion changes, it changes for both on the same day. In Kaizo, editing a criterion’s description applies to every future evaluation, and calibration sessions keep reviewers aligned on edge cases.
How do you compare human and AI agents fairly?
Compare human and AI agents on the shared core, topic by topic, then weight both by the same topic mix. A raw overall average mostly measures which topics each handler gets.
The reason is simple: the bot usually handles the easy topics, and people handle the hard ones. Here is a worked example with numbers we chose to show the arithmetic, not a customer’s data. Scores are the shared core score out of 100:
| Topic | AI agent conversations | AI agent score | Human conversations | Human score |
|---|---|---|---|---|
| Order tracking | 6,000 | 96 | 1,000 | 94 |
| Billing disputes | 1,000 | 70 | 4,000 | 82 |
| Raw average | 7,000 | 92.3 | 5,000 | 84.4 |
On the raw averages, the bot looks 7.9 points better. Topic by topic, the bot is 2 points better on order tracking and 12 points worse on billing disputes.
To compare fairly, weight both by the same mix. Here we use the combined mix: 7,000 order tracking and 5,000 billing conversations out of 12,000.
| Mix-adjusted score | Calculation | Result |
|---|---|---|
| AI agent | (7,000 x 96 + 5,000 x 70) / 12,000 | 85.2 |
| Human team | (7,000 x 94 + 5,000 x 82) / 12,000 | 89.0 |
On the same mix, the human team is 3.8 points better. The decision that follows is not “the bot is better” or “the bot is worse”. It is “keep the bot on order tracking, and keep billing disputes with people until the bot’s billing score improves”. That is the kind of answer a shared core makes possible.
Two more checks keep the comparison honest. Only compare topics where both handlers have enough conversations to be stable, and note it when one handler has almost none. And look at the handler blocks next to the score: a bot with a high core score but failing grounding on one topic still needs a fix.
How to report human and AI quality on one page
Report one page with the shared core score per topic for both handlers, the mix-adjusted totals, and the handler block pass rates beside them. Leadership reads the comparison; the teams read their own blocks.
| Section | Human team | AI agent |
|---|---|---|
| Shared core, mix-adjusted | Score and trend | Score and trend |
| Shared core by topic | Score per topic | Score per topic |
| Handler block | Case notes, follow-up kept: pass rates | Grounding, scope, escalation, uncertainty: pass rates |
| Handoff | Picked up the context | Passed on the context |
| Top causes | Skill gaps, process gaps | Articles, rules, settings |
The bottom row is where the two diverge. A human’s failures usually point to coaching or a process; a bot’s usually point to an article or a setting.
Our post on agent error or process error shows how to tell the two apart before you act. In Kaizo, scoring human and AI conversations on the same scorecard lets you see how both perform side by side.
How to build one scorecard for human and AI agents
Build it in six steps, starting from the scorecard your human team already uses. The first three steps take a working session; the rest happen as you score.
Step 1: Sort your current criteria into three groups
Take your human scorecard and mark each criterion as shared, human only, or not needed. Most teams find that correct answer, policy and next step are shared, while greeting and scripted phrases are human only or can go.
Step 2: Rewrite the shared criteria in neutral words
Rewrite each shared criterion so it does not assume a person or a bot, and so a reviewer can point at the line that passes or fails it.
Step 3: Add the AI agent block
Add the four bot criteria from the template: grounded in your content, scope, escalation trigger and uncertainty. Keep them pass or fail.
Step 4: Fix the weights and the scale for the core
Set one scale and one set of weights for the shared core, adding up to 100, and write them down where both teams can see them.
Step 5: Calibrate on mixed conversations
Run a calibration session on a mix of human and AI conversations, and keep adjusting the criteria descriptions until reviewers agree on both.
Step 6: Score everything and report by topic
Score every conversation, human and AI, on the scorecard, and report the shared core by topic with the mix adjustment. Kaizo’s AutoPilot scores each conversation against your criteria when it resolves, on calls, chat and email.
See it in a live demo. Kaizo scores your team and your AI agents on one scorecard, with the evidence behind every score, and shows both side by side. Let’s talk.
Frequently asked questions
Should the AI agent be held to a higher standard than human agents?
On the shared core, hold both to the same standard, or the comparison means nothing. Where you can be stricter is in the AI agent block and in the consequences: an invented refund can be an auto-fail for the bot, and one critical failure can cost it a topic, because a bot repeats the same mistake across many conversations.
Who should score the human and AI conversations?
The same scorer for both, on the shared core. If people review human conversations and an automated scorer reviews the bot, differences between the two scorers show up as differences between the handlers. Score both automatically on the same criteria, and have reviewers check a mixed slice of both every month.
What if the AI agent scores higher than your human team?
Check the mix first, because the bot usually handles easier topics. If it still scores higher on the same topic after the mix adjustment, that is useful news: it may be ready for more of that topic, and its answers may show where your human team’s process or content needs work.
How many criteria should a shared scorecard have?
Keep the shared core to about six to eight criteria and each handler block to two to four. More than that and reviewers stop scoring consistently, and the comparison gets noisy. If a criterion has not failed in months and drives no decision, remove it or turn it into a pass or fail check.
Can one scorecard cover calls, chat and email?
Yes, if the criteria describe the outcome rather than the channel. A correct answer, policy followed and a clear next step mean the same thing on a call, in a chat or in an email. Add channel-specific checks only where the channel changes what a good conversation looks like, such as verification on calls.
