Multilingual QA means scoring your support conversations in every language you offer against one shared standard, so a 90 in Polish means the same as a 90 in English. It needs one scorecard with a few language-specific criteria, a calibration set per language, and enough coverage that small languages get scored at all.
In short
- With a small QA team, sampling breaks first in the small languages: some queues get no native review at all.
- Most criteria are the same in every language, while register, spelling and local policy need language-specific checks.
- Six steps set it up: one scorecard, language-specific criteria, a calibration set per language, scoring every conversation, cross-language calibration and reporting by language.
- Compare languages only on enough conversations, with a simple rule for when a gap is real.
- Native reviewers, translated reviews and AI scoring each have a place, and most teams combine them.
For the platform side, the page on customer service QA shows how Kaizo scores every conversation against your own criteria.
Why does multilingual QA break with a small QA team?
Multilingual QA breaks because QA analysts can only review languages they read, so a team of two covers two or three languages well and leaves the rest unreviewed. The total sample looks fine while whole queues go unchecked.
Worked example (illustrative numbers, not Kaizo data). A support team handles 40,000 conversations a month in eight languages. Two QA analysts each review 25 conversations a day, 20 days a month: 1,000 reviews, or one conversation in 40. One analyst reads English and German, the other English and French.
| Language | Conversations a month | Agents | Native reviews a month | Reviews per agent a month |
|---|---|---|---|---|
| English | 16,000 | 40 | 400 | 10 |
| German | 8,000 | 20 | 300 | 15 |
| French | 5,000 | 12 | 300 | 25 |
| Spanish | 4,000 | 10 | 0 | 0 |
| Dutch | 2,500 | 6 | 0 | 0 |
| Italian | 2,000 | 5 | 0 | 0 |
| Polish | 1,500 | 4 | 0 | 0 |
| Portuguese (Brazil) | 1,000 | 3 | 0 | 0 |
| Total | 40,000 | 100 | 1,000 | 10 |
On paper, QA covers one conversation in 40. In practice, 11,000 conversations a month, more than a quarter of volume, get no native review, and 28 agents never receive a QA score in their own language. Those agents are coached on nothing, or on a machine-translated reading of their work that nobody checked.
The cost is not abstract. In CSA Research’s 2020 survey of 8,709 consumers in 29 countries, 75% said they were more likely to buy from the same brand again if customer care was in their language. Offering the language is half the job; knowing the support in that language is good is the other half.
Which QA criteria change by language?
Most QA criteria are language-neutral: a correct answer, a followed procedure and a verified identity mean the same in every language. Register, spelling, names and local policy are language-specific and need their own definition per language.
Use this table to sort your scorecard before you translate it:
| Criterion | Same in every language? | What to define per language |
|---|---|---|
| Correct answer | Yes | Nothing, if your knowledge base exists in that language |
| Procedure followed | Yes | Local steps, such as a different returns address or carrier |
| Identity verified | Yes | Nothing, unless checks differ by country |
| Policy limits | Mostly | Local legal terms, such as withdrawal periods or warranty rules |
| Tone and empathy | Mostly | What “warm” sounds like: formal or informal address (Sie or du, vous or tu) |
| Spelling and grammar | No | The language’s own rules and your house style |
| Names and greetings | No | Local name order, titles and greeting conventions |
| Product terms | No | Your approved glossary per language, so agents use the same words as your help center |
Write the language-neutral criteria once.
Then add a short note per language to the criteria marked “Mostly” or “No”, for example: “German: address the customer as Sie unless they wrote du first.” That keeps one scorecard and one score, while each language is judged by its own rules.
Our guide on scoring soft skills in QA shows how to make tone criteria checkable in any language.
How to set up multilingual QA in six steps
To set up multilingual QA, write one scorecard, add language notes, build a calibration set per language, score every conversation, calibrate across languages, and report results by language.
Step 1: Write one scorecard
Start from your current scorecard and keep it as the only one. A separate scorecard per language looks tidy but makes the scores impossible to compare. If you need to rebalance the weights while you do this, see how to weight QA scorecard criteria.
Step 2: Add language notes to the criteria that need them
Use the table above to find the criteria that change by language, and add a one-line note per language. Ask a native speaker on each team, often a senior agent or team lead, to write and check their language’s notes.
Step 3: Build a calibration set per language
Pick 20 to 30 past conversations per language that cover good, average and failed handling, and score them with a native reviewer. This set becomes the reference for that language. Our guide to building a golden set describes how to choose the conversations and agree on the scores.
Step 4: Score every conversation in its own language
Score conversations in the language they were written or spoken in, not in a translation. A translation smooths out exactly what you want to check: register, phrasing and product terms. If you use AI scoring, test it against each language’s calibration set before you rely on it, the way our validation protocol describes.
Step 5: Calibrate across languages
Once a quarter, take five conversations per language, translate them into one shared language for discussion only, and have reviewers from each language score them together.
The goal is not to judge grammar in a translation. It is to check that “correct answer” and “policy followed” are read the same way everywhere. The format for these sessions is in how to run QA calibration sessions.
Step 6: Report by language and by criterion
Show the QA score per language and per criterion, next to volume and the number of scored conversations. A language-level average hides which criterion is behind a gap, and a criterion-level view without volume invites conclusions from ten conversations.
See it in a live demo. Kaizo scores 100% of your conversations against your own criteria, with every score linked to the evidence in the conversation. Let’s talk.
How do you compare quality across languages fairly?
Compare languages criterion by criterion, only where each language has enough scored conversations, and treat a gap as real only when it is larger than what chance would produce on that volume.
A decision rule you can use (example assumptions, check them against your own data):
| Condition | What to do |
|---|---|
| Fewer than 100 scored conversations in a language this month | Do not compare it yet; report the score with its count |
| Gap of less than 5 points on a criterion | Treat as noise; keep watching |
| Gap of 5 points or more, with at least 100 conversations on each side | Read ten failed conversations from the lower language |
| Same criterion low in one language for two months running | Fix the cause: content, process, training or the language note |
Why 5 points and 100 conversations?
If individual QA scores spread with a standard deviation of about 15 points, the average of 100 scores has a standard error of 1.5 points. The gap between two such averages has a standard error of about 2.1 points, so a 5-point gap is about 2.4 standard errors: unlikely to be chance.
With 30 conversations a side, the same gap would be well within noise.
When you read the failed conversations, check the cause before you blame the agents. A low “correct answer” score in one language is often a missing or outdated help center article in that language. A low “procedure followed” score can be a local process nobody documented.
Native reviewers, translation or AI scoring?
Native reviewers give the best judgement but do not scale, machine translation scales but hides register and phrasing, and AI scoring covers every conversation but must be checked against native reviewers per language. Most teams combine all three.
| Method | Strong at | Weak at | Use it for |
|---|---|---|---|
| Native reviewers | Tone, register, phrasing, local expectations | Coverage: one reviewer per language rarely fits the budget | Calibration sets, disputes, coaching in the agent’s language |
| Machine-translated review | Letting one analyst read any language | Grammar, register and product terms disappear in translation | Checking language-neutral criteria only |
| AI scoring in the original language | Scoring every conversation, every language, the same way | Needs testing per language before you trust it | Coverage of all languages, flagging what a native reviewer should read |
| Outsourced native reviewers | Adding a language quickly | Calibration with your own team takes work | Large languages without in-house reviewers |
Kaizo’s site says its sentiment analysis works across the languages your team supports, and that its coaching cards are available in every major support language. Foot Locker uses Kaizo across multiple European languages. For the exact languages Kaizo scores and how criteria apply per language, ask in the demo for your own language mix.
How to start multilingual QA
Start with your smallest languages, not your largest: list every language, its volume and how many conversations got a native review last month. The languages with zero are your first gap, and a calibration set of 20 conversations for each is your first step.
To see how Kaizo applies one standard to every conversation, go to Kaizo’s QA platform.
Frequently asked questions
Can one QA scorecard work in several languages?
Yes, and it should. Keep the criteria that mean the same thing everywhere, such as correct answer and procedure followed, and add a short note per language where the rules differ, for example on formal address or spelling. One scorecard keeps the scores comparable across languages and teams.
Do QA reviewers need to speak the language they score?
For tone, register and spelling, yes: those cannot be judged reliably through a translation. For language-neutral criteria, such as whether the answer was correct or the procedure followed, a reviewer can work from a translation. Use native reviewers for calibration and disputes, even if most scoring runs another way.
How do you QA a language nobody on the QA team speaks?
Build a calibration set with a native speaker, often a senior agent from that team, and use it to check whatever scores that language, whether a translated review or AI scoring. Then have the native speaker review the failed conversations each month, which takes far less time than reviewing a sample.
Should agents be scored on grammar in their second language?
Score grammar where it affects understanding or your brand, and say so in the language note. Many teams weigh spelling and grammar lower than a correct answer and the right procedure, and score errors that change the meaning, not minor slips that a customer would not notice.
How many conversations per language do you need to compare QA scores?
As a working rule, at least 100 scored conversations per language per period before you compare averages, and more for criteria that rarely apply. Below that, a few unusual conversations can move the average by several points, so report the score with its count and wait.
