Traditional QA cadence was a sampling plan, a few conversations per agent per week, sized to what reviewers could read. Once every conversation is scored, cadence becomes how often you act on what you already see.
In short
- A few conversations per agent can’t support agent-level decisions.
- Keep sampling for calibrating reviewers or the automated grader.
- Layer it: continuous scoring, weekly pattern review, periodic calibration.
- Add event-triggered deep dives on top.
Cadence is usually a sampling plan wearing a schedule
Ask most teams what their QA cadence is and you get a sampling plan: so many conversations per agent per week, reviewed by the QA team on a set day. That framing feels like a schedule, but it is really a statement about capacity. The number of conversations was chosen because it was as many as the reviewers could get through, and the frequency was chosen to fit the roster. Neither number was derived from what the decisions being made actually require.
That is the quiet problem. A cadence built around reviewer capacity produces just enough evidence to feel like oversight and not enough to support the conclusions drawn from it. The fix is not a better sampling plan. It is to separate the two jobs the sample was doing: measuring agents, and checking the measurement.
Why a small sample cannot carry agent-level decisions
Suppose you review a handful of conversations per agent each week. It feels reasonable, and for a rough sense of the team it is. But the moment you use it to compare two agents, rank them, or make a decision about one person, the sample is being asked to do something it cannot.
The reason is statistical, and it does not require a formula to feel. A few conversations is a tiny window on someone’s month, and which conversations landed in the window matters enormously. Two agents of genuinely equal quality can post very different sampled scores simply because of which handful got pulled, and a real difference between two agents can vanish or reverse depending on the draw. The sample error is large next to the differences you are trying to detect, so the ranking you build on it is mostly noise. Our note on measuring reviewer agreement covers the related problem of whether two reviewers even score the same conversation the same way.
This is not an argument that sampling is worthless. It is an argument that a sample sized for reviewer capacity is the wrong instrument for an agent-level decision, and that pretending otherwise is how QA loses the trust of the floor.
What changes when you can score everything
Automated scoring removes the capacity constraint that made sampling necessary. When every conversation is scored, the agent-level decision is no longer built on a draw of a few conversations; it is built on all of them. That does two things to cadence.
First, it retires the sampling plan as the definition of your cadence. You are no longer deciding how many to read. Second, it moves the remaining sample to a different job: checking the scorer. You still pull a set of conversations at a regular interval, but now it is to verify that the automated grader agrees with your reviewers, which is the subject of validating AI QA scoring. The sample stops being how you measure agents and becomes how you keep the measurement honest. Coverage itself is covered in how to score 100% of conversations.
A monitoring rhythm designed around coverage
With scoring continuous, the useful cadence is a set of layers, each answering a different question.
| Layer | Rhythm | What it answers |
|---|---|---|
| Continuous scoring | Every conversation, as it closes | What is the current state of quality, without waiting for a review day |
| Pattern review | Weekly | What is trending, which criteria are slipping, where coaching should go |
| Calibration and grader check | Periodic | Do reviewers and the automated scorer still agree on the same conversations |
| Event-triggered deep dive | On a spike, a complaint, or a policy change | What happened, across all affected conversations rather than a sample |
Cadence for AI agents specifically
For AI agents the case for coverage over sampling is even stronger, because their failures are systematic. A human’s mistakes are scattered across a month, so a sample has a fair chance of catching a representative few. An AI agent’s mistakes cluster around a specific prompt weakness or a specific type of request, so a sample sized for humans can miss an entire failure mode until it is already widespread, as covered in the silent-failure taxonomy. The right cadence for automation is continuous scoring with event-triggered review keyed to prompt or knowledge-base changes, because a change to the agent can change its failure profile overnight.
One structural note on who runs the cadence. If the platform doing the continuous scoring is also the one that sold the AI agent, its monitoring has a reason to be gentle at exactly the moments that matter. Kaizo does not sell its own AI agents, so its scoring runs on the evidence regardless of what it finds, and each finding traces to the transcript, which is what makes an event-triggered deep dive actionable rather than defensive.
Frequently asked questions
How often should you run QA reviews?
The honest answer is that review frequency is the wrong question once scoring can be continuous. Traditional cadence, a few conversations per agent per week, was shaped by how many a human could read, not by what the decision needs. With automated scoring, the useful rhythm is continuous scoring plus a weekly pattern review, a periodic calibration, and event-triggered deep dives.
How many conversations should you sample for QA?
For measuring individual agents, a sample sized to reviewer capacity is too small to support the conclusion, because the sample error is large relative to the differences you are trying to detect. Sampling remains valid for a different job: calibrating reviewers and checking that an automated grader agrees with them. For agent-level decisions, full coverage is the reliable basis.
Does full QA coverage remove the need for calibration?
No. It changes what the sample is for. Instead of sampling conversations to score agents, you sample them to check the scorer, verifying that reviewers and the automated grader still agree. Calibration becomes more important with automation, not less, because a single grader now influences every score.
What monitoring cadence suits AI agents?
Continuous scoring with event-triggered review keyed to prompt or knowledge-base changes. AI agent failures are systematic rather than random, so they cluster around specific weaknesses and a human-sized sample can miss a whole failure mode. A change to the agent can shift its failure profile overnight, which is why review should be triggered by changes, not only by the calendar.
Related terms
- How to score 100% of conversations
- How to validate AI QA scoring
- How to measure reviewer agreement
- Silent failures in AI agents
- How to run QA calibration sessions
Design a QA cadence around coverage, not capacity
Tell us how you sample QA today and what decisions you make from it. We will show you what full coverage would change, which of your current conclusions the sample cannot actually support, and how a layered monitoring rhythm looks when every conversation is already scored. Every score traces back to the evidence in the transcript.
