QA methodologies in customer service are the rules for which conversations get reviewed, by whom and how often. The five main ones are random sampling, targeted review, risk-based review, calibration-led review and 100% coverage. Each finds different problems and misses others, so most mature teams combine two or three of them in one QA plan.
In short
- A QA methodology decides which conversations get reviewed; the scorecard decides how each one is judged.
- Random sampling is fair and simple but sees too little to judge one agent; targeted and risk-based review find problems faster but are biased by design.
- Calibration-led review is about making scores consistent, not about finding more problems.
- 100% coverage scores every conversation automatically, and people review where their judgement adds something.
- A worked QA plan for a 300-agent team shows how to combine the methods within real reviewer capacity.
This page compares the methods. If you are setting up QA from zero, start with our 7-step QA program playbook; if you want the basics of sampling, read what QA sampling is. Kaizo is our product and it supports full coverage. We say below where the other methods do the job better.
Which QA methodology fits which job?
Pick the methodology by the question: random sampling for a fair average, targeted for a known problem, risk-based for costly mistakes, calibration-led for consistency, full coverage for per-agent patterns.
| Methodology | How conversations are picked | What it is good at | What it misses | Reviewer effort | Best for |
|---|---|---|---|---|---|
| Random sampling | A fixed number per agent or queue, picked at random | An unbiased average; easy to defend as fair | Rare and agent-level problems; most conversations are never seen | Medium, grows with headcount | Small teams; a baseline check on any other method |
| Targeted review | Filters for a known issue: a topic, queue, new hires, low CSAT | Finding and fixing a specific problem fast | Everything outside the filter; scores skew low | Low to medium | Launches, new hires, a known complaint spike |
| Risk-based review | Rules or scores flag conversations where a mistake is costly | Compliance, refunds, vulnerable customers, escalations | Quiet problems that do not trip a flag | Medium | Regulated and high-value queues |
| Calibration-led review | The same conversations scored by several reviewers, then compared | Consistent scoring; a shared meaning for each criterion | Coverage; it reviews few conversations by design | Low in volume, high in time per conversation | Any team with more than one reviewer |
| 100% coverage | Every conversation scored automatically, people review flags and disputes | Per-agent patterns, rare issues, trends across all contacts | Nuance the scorecard does not capture, unless people review | Low per conversation, setup and validation up front | Teams with volume beyond what reviewers can read |
The sections below take each method in turn, with a verdict, how it works and where it breaks.
What is a QA methodology in customer service?
A QA methodology in customer service is the agreed way of choosing, reviewing and acting on conversations to check quality. It sits next to the scorecard: the scorecard says what good looks like, and the methodology says which conversations you check against it and what happens next.
Teams write it down in a QA plan. A QA plan names the scorecard, the methodology or mix of methodologies, how many conversations each person reviews, who calibrates and how often, how agents dispute a score and how results turn into coaching.
Without a written plan, the method becomes whatever each reviewer has time for. The plan template further down covers each part.
Random sampling
Verdict: best as a fair baseline and as a check on other methods; too thin to judge one agent. Random sampling reviews a fixed number of randomly chosen conversations per agent or queue, usually a few per week.
Its strength is fairness. When conversations are truly random, an agent cannot say QA only picks their worst calls, and the team-wide average is an honest estimate. Our post on QA call selection explains why manual “random” picks are often not random at all.
Its weakness is arithmetic. Here is a worked example of what a sample can tell you, using standard 95% confidence intervals:
| What was reviewed | Observed pass rate on “correct answer” | Range the true rate is likely in |
|---|---|---|
| 5 conversations of one agent in a month (4 passed) | 80% | about 38% to 96% |
| 20 conversations of one agent (16 passed) | 80% | about 58% to 92% |
| 400 conversations across a team (340 passed) | 85% | about 81% to 88% |
At five reviews a month, an 80% agent and a 50% agent can look identical. At team level, the same method is fine. Our post on why a 3% QA sample cannot support agent-level decisions goes deeper into the statistics.
Targeted review
Verdict: the fastest way to fix a known problem; a poor measure of overall quality. Targeted review filters conversations by a condition you already care about, such as a topic, a queue, new hires, low CSAT or reopened tickets, and reviews those.
Use it when you have a question: is the new refund policy being explained correctly, how are this month’s new hires handling cancellations, why did CSAT drop on the billing queue. You will find the answer much faster than with a random sample.
The catch is that targeted scores are not comparable with random ones. If you review low-CSAT conversations, scores will be lower than average, and an agent who handles more of those queues will look worse. Report targeted reviews separately and never mix them into an agent’s average.
Risk-based review
Verdict: essential where one mistake is expensive; blind to anything the rules do not flag. Risk-based review sends the conversations most likely to cause harm to a person first, flagged by rules or by an automated score.
Typical flags are refunds over a threshold, cancellations, complaints, legal or regulatory topics, vulnerable customers, escalations, policy exceptions and auto-fail criteria such as sharing payment details. Our guide to auto-fail criteria covers how to define the most serious ones.
Risk-based review only finds what you thought to flag. A new kind of mistake, or a slow drift in tone, will not trip a rule. That is why it works best on top of a broader method, either random sampling or full coverage, rather than alone.
Calibration-led review
Verdict: not a way to find problems, but the method that makes every other method trustworthy. Calibration-led review has several reviewers score the same conversations, compare the differences and agree what each criterion means.
A typical session takes a handful of edge cases, the conversations reviewers argue about, and has everyone score them against the same scorecard. Where scores diverge, the team agrees the meaning and writes it into the criterion description.
Our guide on running QA calibration sessions has the routine, and a golden set of agreed scores gives you a fixed reference.
Some teams run their whole QA program calibration-first: a smaller number of conversations, each reviewed by two people, with every disagreement resolved. That gives very reliable scores on few conversations. It is a good choice when scores feed pay or performance decisions and volume is low.
100% coverage
Verdict: the only method that sees every agent’s full record and rare issues; it needs validated scoring and people on the conversations that matter. With 100% coverage, every conversation is scored automatically against your scorecard, and human reviewers work on the flags, disputes and edge cases.
The change is in what QA can answer. With every conversation scored, a criterion missed in 2% of one agent’s conversations is a number, not a guess. Trends across topics, queues and channels show up without anyone choosing where to look. Our explainer on 100% QA coverage covers the definition.
The risks are real, too. Automated scoring is only as good as the scorecard and the instructions behind it, so test it against conversations your team has already rated before you trust it. And people still need to review the conversations where judgement matters.
In Kaizo’s customer service QA, AutoPilot scores every conversation against your own scorecard once it resolves, with a comment that explains each score.
Assignments then send human reviewers to the conversations where a person adds something, by queue, channel or agent, or by signals such as negative CSAT and high reopens, and Auto QA pre-fills those reviews for the reviewer to edit and approve.
How do you combine QA methodologies in one plan?
Combine QA methodologies by giving each a job: 100% coverage or random sampling for the baseline, risk-based review for costly mistakes, targeted review for this month’s question, and calibration to keep reviewers aligned. Then fit the plan to your real reviewer hours.
Here is a worked example for a 300-agent team handling 150,000 conversations a month with six QA analysts. The numbers are illustrative.
Step 1: Work out reviewer capacity
Six analysts with 120 review hours each per month, at about 10 minutes per review, can do 4,320 reviews a month. That is 2.9% of all conversations, or about 14 per agent per month, if they did nothing else.
Step 2: Decide what covers the baseline
With sampling only, those 4,320 reviews are the baseline and almost nothing is left for risk or targeted work. With full coverage, all 150,000 conversations are scored automatically, and the 4,320 human reviews go where people add judgement.
Step 3: Allocate the human reviews
| Job | Methodology | Human reviews per month | Why this many |
|---|---|---|---|
| High-risk flags: refunds, complaints, auto-fails | Risk-based | 1,500 | Every flagged conversation in the riskiest queues |
| New hires and this month’s focus topic | Targeted | 1,200 | About 40 per new hire, plus the focus topic |
| Random check of automated scores | Random sampling | 600 | Enough for a team-level accuracy check each month |
| Agent disputes | Dispute review | 300 | Based on the dispute rate you expect |
| Calibration: 20 conversations by each of 6 analysts | Calibration-led | 120 | Monthly session plus new edge cases |
| Spare for coaching preparation | 600 | Buffer for peaks and deep dives | |
| Total | 4,320 |
The random check is the line teams forget. It keeps the automated scoring honest, in the same way a random sample keeps targeted reviews honest. Our guide on QA monitoring cadence covers how often each review type should run.
What goes in a QA plan?
A QA plan is a one-page document that names the scorecard, the methodology mix, review volumes, calibration routine, dispute process and how findings turn into coaching and fixes. Copy the template below and fill in each line.
| Section | What to write | Example |
|---|---|---|
| Scope | Channels, queues and teams covered | Chat and email, all queues, in-house and outsourced |
| Scorecard | Name, version, owner | Support scorecard v3, owned by QA lead |
| Baseline method | How overall quality is measured | All conversations scored automatically; 600 random human checks a month |
| Risk rules | What gets flagged for human review | Refunds over $200, complaints, auto-fail criteria |
| Targeted focus | This month’s question | New cancellation policy; new hires in weeks 1 to 6 |
| Calibration | Who, how often, how many conversations | All analysts, monthly, 20 conversations |
| Disputes | How agents challenge a score, response time | In the tool, answered within five working days |
| Reporting | What goes to whom | Agent results to team leads weekly; trends to the head of support monthly |
| Actions | How findings become coaching or fixes | Skill gaps to coaching; process gaps to the owner of the macro or policy |
| Review date | When the plan itself is revisited | Every quarter, or after a policy change |
If you are separating QA from QC in the same plan, our explainer on quality assurance vs quality control shows where each sits.
How to choose your QA methodology mix
Choose your mix by volume and risk: small teams can run random sampling plus calibration, risky queues add risk-based review, and large teams move the baseline to full coverage.
Once volume outgrows reviewer capacity, people work on risk, targets and disputes rather than the baseline.
Use this decision rule:
- Under about 20 agents, low risk: random sampling of a fixed number per agent, monthly calibration, targeted review when something changes.
- Any size, regulated or high-value contacts: add risk-based review of every flagged conversation in those queues.
- Scores feed pay or performance reviews: add double review on the conversations that count, and a dispute process.
- Reviewer capacity under about 5% of volume: move the baseline to 100% coverage, validate the scoring, and keep a random check on it.
See a QA plan on full coverage. Kaizo scores every conversation against your scorecard and sends reviewers to the ones that need a person. Book a demo to see it on realistic sample data.
Frequently asked questions
What is the most common QA methodology in customer service?
Random sampling is still the most common, usually a fixed number of conversations per agent per week or month, often combined with targeted review of complaints or low CSAT. Teams with high volume increasingly score every conversation automatically and keep human review for risk, disputes and calibration.
What is the difference between a QA methodology and a QA process?
The methodology is the rule for choosing conversations: random, targeted, risk-based, calibration-led or full coverage. The process is everything around it: scoring against the scorecard, giving feedback, handling disputes, coaching and fixing causes. You can change the methodology without rebuilding the whole process, which is how most teams move from sampling to full coverage.
How many conversations should QA review per agent?
It depends on what the reviews are for. A few per month gives a rough team average but cannot reliably show one agent’s quality. Twenty or more per agent narrows the range considerably. When scores feed performance reviews, score far more, or all of them, automatically.
Do you still need random sampling with 100% coverage?
Yes, in a smaller role. With 100% coverage, a random set of automatically scored conversations reviewed by people each month checks that the scoring still matches your standard. It catches drift after a scorecard or policy change that risk rules and disputes would not show.
What is risk-based QA?
Risk-based QA reviews the conversations where a mistake is most costly first, such as refunds, cancellations, complaints, regulatory topics or vulnerable customers. Rules or automated scores flag them for a person. It is a priority layer, not a full method, because it only catches the risks someone thought to flag.
