QA calibration software is the part of a quality assurance tool that gets reviewers scoring the same conversation the same way. It lets several reviewers score one conversation, shows where and by how much they differ, tracks each reviewer’s drift over time and writes the agreed decision back into the scorecard, so an agent’s score depends on the conversation, not on who reviewed it.
In short
- QA calibration software replaces the shared spreadsheet with independent scoring, automatic comparison and a record of each decision.
- Each common reviewer bias, from leniency to drift, has a feature that counters it; the table below maps them.
- The buyer checklist lists ten features, five of them must-haves for a team with several reviewers.
- The worked example shows how to measure agreement and reviewer offset on one criterion, and why one reviewer can move an agent’s score by a full point.
- AI scoring applies the rubric the same way every time, which removes reviewer-to-reviewer variation but not a badly written rubric.
- A decision rule tells you when a spreadsheet is still enough.
This guide is for choosing the tool. For the method itself, read what QA calibration is and our playbook on how to run QA calibration sessions. If you are comparing customer service QA software, calibration is one of the features that separates a scoring tool from a fair one.
What QA calibration software does that a spreadsheet does not
QA calibration software runs the same conversation past several reviewers independently, compares their scores per criterion, and keeps the decision with the scorecard. A spreadsheet can hold the scores, but it relies on someone to keep reviewers blind, do the maths and update the rubric.
| Job in calibration | In a shared spreadsheet | In calibration software |
|---|---|---|
| Pick the conversations | Copied links, often the easy cases | Selected inside the tool, including edge cases |
| Score independently | Reviewers can see each other’s columns | Scores hidden until everyone has scored |
| Compare | Someone builds the formulas | Spread per criterion shown automatically |
| Track reviewers over time | Rarely done | Each reviewer’s offset from the group, session after session |
| Record the decision | Meeting notes | Written into the criterion description |
| Apply it to future scores | Depends on memory | Every later evaluation uses the updated criterion |
The last two rows matter most. A calibration that ends in meeting notes produces agreement for a week; one that ends in the criterion keeps it.
Where reviewer bias in QA scores comes from
Reviewer bias comes from people, not bad intent: some reviewers score high or low by habit, cluster in the middle, judge the agent instead of the conversation, or drift as months pass. Each pattern has a feature that counters it.
| Bias | What it looks like | Feature that counters it |
|---|---|---|
| Leniency or severity | One reviewer is always above or below the group | Reviewer offset report against the group or a baseline score |
| Central tendency | Everything gets a 3 out of 5 | Spread per criterion, and anchors for each point of the scale |
| Halo effect | A strong opening lifts every other criterion | Criterion-level scoring with evidence for each line |
| Familiarity | Known high performers get the benefit of the doubt | Hiding the agent’s name during calibration (ask whether the tool can) |
| Anchoring | Later reviewers copy the first score they see | Blind scoring until all reviewers have submitted |
| Drift | A reviewer’s standard moves over months | Offset tracked over time, and regular sessions |
| Workload | Tired reviewers score the end of the queue differently | Even distribution of review work across reviewers |
| Selection | Calibrating only easy conversations hides disagreement | Choosing edge cases on purpose |
Calibration fixes disagreement between reviewers. It does not, by itself, fix a criterion that measures the wrong thing; that is a scorecard decision.
A buyer checklist for QA calibration software
Look for ten features, and treat the first five as must-haves if more than two people score conversations. Ask each vendor to show them in a demo rather than tick them on a form.
| # | Feature | Must-have? | What to ask the vendor to show |
|---|---|---|---|
| 1 | Several reviewers score the same conversation in a session | Yes | A session set up with three reviewers on the same ticket |
| 2 | Scores stay hidden until everyone has scored | Yes | What a reviewer sees before and after submitting |
| 3 | Spread shown per criterion, not only per conversation | Yes | Where the biggest disagreement is in one session |
| 4 | The decision written into the criterion description | Yes | A criterion edited after a session, and when the change applies |
| 5 | Every score linked to evidence in the transcript | Yes | The line in the conversation behind a deduction |
| 6 | Reviewer offset tracked across sessions | Strongly advised | One reviewer’s trend over several months |
| 7 | Even distribution of review work | Advised | How assignments are spread across reviewers |
| 8 | A dispute flow for agents | Advised | An agent challenging a score, and where it lands |
| 9 | AI scoring as an extra rater, with agreement per criterion | If you automate | AI scores against your calibrated reviewers on the same set |
| 10 | Agent names hidden during calibration | Optional | A session view without the agent’s name |
Feature 9 is where calibration and automation meet. Before you trust automated scores, you need a set of conversations your calibrated reviewers agree on, as described in our guide to building a golden set.
How to measure scoring fairness: a worked example
Measure fairness on each criterion with three numbers: how often reviewers agree exactly, how often they are within one point, and how far each reviewer sits from the group average. This illustrative example uses made-up scores for one criterion.
Three reviewers score “Empathy” on a 1 to 5 scale for the same six conversations.
| Conversation | Reviewer 1 | Reviewer 2 | Reviewer 3 | Spread (highest minus lowest) |
|---|---|---|---|---|
| 1 | 4 | 3 | 4 | 1 |
| 2 | 3 | 2 | 4 | 2 |
| 3 | 5 | 4 | 5 | 1 |
| 4 | 2 | 2 | 3 | 1 |
| 5 | 4 | 3 | 4 | 1 |
| 6 | 3 | 3 | 3 | 0 |
| Average | 3.50 | 2.83 | 3.83 | 1.0 |
Step 1: Calculate exact and within-one agreement
All three reviewers gave the same score on one conversation of six, so exact agreement is 1 in 6. They were within one point on five of six, so within-one agreement is 5 in 6. That gap usually means the scale has more points than the criterion’s description can separate.
Step 2: Calculate each reviewer’s offset
The group average across all 18 scores is 61 ÷ 18 = 3.39. Reviewer 1 sits at +0.11, Reviewer 2 at -0.56 and Reviewer 3 at +0.44. Reviewer 2 is the lowest or joint lowest on all six conversations, which is a severity pattern rather than chance.
Step 3: Translate it into what an agent feels
If Reviewer 2 reviews one agent and Reviewer 3 another, two agents with identical conversations end up a full point apart on a five-point scale: 2.83 against 3.83. That is the unfairness agents notice, and it comes entirely from who opened the ticket.
Step 4: Fix the criterion, then measure again
Write an anchor for each point of the scale with an example, discuss conversation 2 where the spread was largest, and score a fresh set at the next session. Track the three numbers per criterion over time; the existing playbook covers how to measure whether calibration is working in more depth.
Can AI remove human bias from QA scoring?
AI removes the variation between reviewers because it applies the same rubric the same way to every conversation, but it does not remove bias written into the rubric itself. Consistency is necessary for fairness; it is not the same thing.
Three things decide whether AI scoring is fair as well as consistent:
- The rubric: a vague or loaded criterion is applied consistently and wrongly. Calibrating your human reviewers first is how you find those criteria.
- The test: compare AI scores with a set your calibrated reviewers agree on, criterion by criterion, before go-live. Our protocol to validate AI QA scoring gives a one-week plan.
- The appeal: agents need a way to dispute a score and see the evidence, as in our guide to designing an appeal process for AI QA scores.
With AI scoring every conversation, calibration changes job. It stops being the way to police reviewers on a sample and becomes the way to agree the standard that the AI then applies everywhere.
Spreadsheet or calibration software: a decision rule
A spreadsheet is enough while one or two people score and the scorecard is short; software pays off once three or more reviewers score, or when scores feed agent decisions. Use the table to place your team.
| Your situation | Spreadsheet enough? | Why |
|---|---|---|
| One or two reviewers, under 10 criteria | Yes | Few pairs to compare, disagreement is easy to see |
| Three or more reviewers | No | Blind scoring and spread per criterion get hard to manage by hand |
| Several sites, teams or clients | No | Offsets between groups need tracking over time |
| Scores feed pay, promotion or performance decisions | No | You need evidence for each score and a record of each decision |
| AI scores part of your volume | No | You need agreement per criterion between AI and your reviewers |
If two or more rows apply, the time spent building formulas and chasing blind scores costs more than the tool. Our post on when to stop running QA in a spreadsheet covers the wider move.
How Kaizo handles calibration
In Kaizo, calibration runs a conversation past several reviewers, shows where they diverged, and writes the agreed meaning into the criterion description. AutoPilot then applies the updated description to every future evaluation.
Here is what that covers from the checklist, based on Kaizo’s product pages:
- Sessions with several reviewers scoring the same conversations against the same scorecard (features 1 and 3).
- Decisions written into the rubric, so they do not live in someone’s memory, and new reviewers are onboarded against the agreed standard (feature 4).
- Reviewer drift made visible instead of invisible (feature 6).
- Evidence for every score: each score links to the evidence in the conversation (feature 5).
- Assignments that distribute review work evenly across reviewers (feature 7).
- AI checked against your team: you test the AI against your historical ratings before it goes live, and AutoPilot interprets the rubric identically every time, so calibration is for the human reviewers who work alongside it (feature 9).
How to start making QA scoring fair this month
Start with one criterion your reviewers argue about, five or six edge-case conversations and three reviewers scoring blind. Calculate agreement and offsets as in the worked example, rewrite the criterion, and repeat at the next session.
Do that for your three most disputed criteria before you compare tools. You will know which features you need, and you will have the reference set to test any vendor’s AI scoring against later.
See a calibration session in a live demo. We show how reviewers’ scores are compared and how the decision updates the rubric, on realistic sample data. Let’s talk.
Frequently asked questions
What is inter-rater reliability in QA?
Inter-rater reliability is how consistently different reviewers give the same score to the same conversation. In QA it is usually measured per criterion as the share of exact agreement, the share within one point, or a statistic such as Cohen’s kappa that corrects for chance agreement. Low reliability means a score says as much about the reviewer as the agent.
How do you reduce bias in QA scoring?
Score blind, compare per criterion, and fix the wording of the criteria that keep causing disagreement. Track each reviewer’s offset from the group over time, spread review work evenly, and require evidence from the conversation for every deduction. Give agents a way to dispute a score, and feed disputes into the next calibration session.
Should agent names be hidden during QA calibration?
Hiding agent names during calibration reduces familiarity bias, where reviewers judge a known high or low performer instead of the conversation. It matters most in small teams where reviewers know every agent. The person running the session can keep the names, so feedback can still reach the right agent afterwards.
What is a baseline score in QA calibration?
A baseline score is a reference score for a calibration conversation, usually set by the QA lead or agreed by the group, that each reviewer’s score is compared against. It turns a debate into a measurement: you can see who sits above or below the baseline and on which criteria.
Can you run QA calibration in Excel?
Yes, for one or two reviewers and a short scorecard. Each reviewer scores in a hidden column, and simple formulas calculate agreement and the spread per criterion. It stops working when reviewers can see each other’s scores, when there are several teams to compare, or when the decisions never make it back into the scorecard.
