Skip to content

Quality Assurance

Same conversation, same score.

Calibration sessions get your reviewers scoring edge cases the same way, so a difference in a score means a difference in the conversation rather than in who reviewed it.

Sessions

Disagreement, surfaced and settled.

Run a conversation past multiple reviewers, compare where they diverged, and agree what the criterion actually means. The outcome updates the rubric rather than living in someone’s memory.

  • Reviewer drift visible instead of invisible
  • Edge cases resolved into the criteria
  • New reviewers onboarded against an agreed standard

How it works

How a calibration session runs, in four steps

  1. Pick the edge cases

    Choose a few conversations that sit on the edge of a criterion: the ones reviewers argue about, or that a new reviewer is likely to score differently.

  2. Run them past several reviewers

    Each reviewer scores the same conversations against the same scorecard.

  3. Compare where scores diverge

    Look at where reviewers diverged and talk through why. The disagreement usually comes down to what one criterion actually means.

  4. Write the decision into the rubric

    Agree the meaning and update the criterion description, so the outcome lives in the rubric rather than in someone's memory. AutoPilot applies the new description to every future evaluation.

Worked example

One criterion, three reviewers

An illustration of a typical session, not a customer case.

  1. The criterion

    "Agent confirms the issue is resolved before closing the ticket."

  2. The disagreement

    Two reviewers pass a chat because the agent asked "Is there anything else?". The third fails it because the customer never confirmed the fix.

  3. The decision

    The team agrees that a pass needs the customer's confirmation, or a follow-up when the customer does not reply, and adds that to the criterion description.

  4. The result

    The next time the case comes up, every reviewer scores it the same way, and new reviewers learn the rule from the rubric.

Go deeper: what QA calibration is, how to run calibration sessions that remove scoring bias and how to build a golden set to check scoring against.

FAQ

Frequently asked questions

Does AutoPilot need calibrating?

No. It interprets the rubric identically every time, which is the point. Calibration is for the human reviewers who work alongside it.

How often should we calibrate?

Most teams run a session monthly, and whenever a standard changes materially.

Should an agent appeal a score caused by a system error?

Yes. If a tool or process failure caused the miss, the score says nothing about the agent. Raise it with the reviewer and bring the case to the next calibration session; if the team agrees, add a note to the criterion description on how to score that situation, so it is not marked against an agent again.

Who should take part in calibration?

Everyone who scores conversations: QA reviewers, team leads who review, and new reviewers, who are onboarded against the agreed standard.

Security & compliance

SOC 2 Type II, ISO 27001 and GDPR

Audit reports, certificates and policies in one place.

Trust Center ›

Find out how far apart your reviewers are

We will run the same conversations past your team and show you where the scores diverge.

Trusted by global support teams

  • Foot Locker
  • Canva
  • GetYourGuide
  • Instacart
  • BlaBlaCar
  • UiPath