To QA the Fin AI agent, read and score a planned sample of Fin conversations against your own scorecard, using the evidence Intercom already gives you: the answer debugger, the sources Fin found, ratings and escalation logs. Fin’s resolution rate tells you the customer left without asking for more help. It does not tell you the answer was right.
In short
- Intercom counts a Fin resolution when the customer confirms the answer helped or leaves without asking for more help, so a resolution is a billing event, not a quality verdict.
- Fin’s CX Score is built for conversations with at least two customer replies and two agent or bot replies, so many one-question Fin conversations get no rating.
- Sample in two parts: a random sample of resolved conversations to correct the resolution rate, and targeted reviews of risky topics, escalations and bad ratings.
- Check each conversation against a fixed checklist: answer accuracy against the source, correct resolution, procedure completed, handover, loops, tone and policy.
- Every failure ends as one change in Fin (content, audience, guidance, procedure or escalation rule), tested in a preview or eval before it goes live.
This is the Fin version of our general guide on how to QA AI agents, and it follows the same steps as our guide to QA for Zendesk AI agents. If you are still choosing how to measure the agent before launch, start with AI agent evaluation, which separates pre-launch evals from scoring live conversations.
What does Fin’s own reporting tell you about quality?
Fin’s reporting tells you volume, coverage and sentiment: how many conversations Fin joined, how many it resolved, how customers rated them and which topics drive the volume. It does not tell you, conversation by conversation, whether an answer was accurate, allowed by policy or complete. For that you need a review against a written standard.
Here is what each Intercom feature gives a QA lead, as described in Intercom’s help center on November 13, 2026:
| Where in Intercom | What it tells you | What it does not tell you |
|---|---|---|
| Performance dashboard (Fin AI Agent > Analyze > Performance) | Automation rate, involvement rate, resolution rate, and links to the conversations behind each path | Whether a resolved conversation had a correct answer |
| Fin CSAT (conversation ratings) | The share of positive ratings for Fin, from the customers who answer the survey | What the silent majority thought, or why a rating was given |
| CX Score (Pro add-on) | A 1 to 5 rating from a machine learning model, built from reasons such as answer quality and customer effort | Anything about short conversations that do not meet its eligibility rules |
| Topics Explorer and Recommendations (Pro add-on) | Topics and subtopics behind the volume, and weekly suggestions to fix gaps in Fin’s content | Whether Fin’s answers on a well-covered topic are right |
| Monitors and Scorecards (Pro add-on) | Rule-based selection of conversations to review, scored against criteria you define | Whether the AI scoring agrees with your reviewers, unless you test it |
| Answer debugger (“Improve answer” in the Inbox) | The question, Fin’s answer and the content items Fin found relevant | Whether those items are correct or current |
Two details matter for QA. First, the CX Score only rates a conversation that is closed, runs over chat or email, was started by the customer, and has at least two customer replies and two agent or chatbot replies. Intercom says it has broadened the criteria so some short conversations with a clear signal may now be scored, but a customer who asks one question, gets one answer and leaves will often go unrated, and those are exactly the conversations that count as assumed resolutions.
Second, Intercom changed how the dashboard counts. Since July 2026 (rollout July 1 to 8), conversations where Fin was active but had no chance to answer (“Fin Constrained”) no longer count as Fin involved, so resolution rates went up and involvement rates went down without any change in quality or in the number of resolutions. If you compare months across July 2026, note the break in the series.
How does Fin count a resolution, and why is it not a quality score?
Intercom counts a resolution when, after Fin’s last answer, the customer either confirms the answer helped (a confirmed resolution) or leaves without asking for more help (an assumed resolution). It is billed as an outcome. Neither path checks whether the answer was correct, so a customer who gave up counts the same as a customer who was helped.
From Intercom’s article on Fin outcomes, fetched November 13, 2026:
- A resolution is only counted when Fin gives an actual answer, and you are charged for one outcome per conversation.
- Resolutions, procedure handoffs and some other outcomes are listed at $0.99 each for Fin over chat and email.
- If the customer comes back to the same conversation for more help, even in a later billing period, the resolution is deducted.
- If the customer explicitly asks for a human, or Fin escalates through its default behavior such as detecting frustration, it is not an outcome and is not billed.
The definition covers a customer returning to the same conversation. It does not cover one who opens a new conversation about the same problem, calls, or churns. Those are the cases a QA review has to catch.
That is why an audit puts a second number next to Fin’s: the share of resolved conversations that were actually correct. Our guide to the AI agent resolution audit explains the backend checks and the repeat window, and our page on correct resolution rate explains how to define and report the corrected figure.
How do you sample Fin conversations for review?
Sample Fin conversations in two parts. A random sample of resolved conversations gives you an honest correct-resolution rate. Targeted reviews of risky topics, escalations, negative ratings and reopened conversations find the failures that cost the most. Keep the two apart: targeted reviews are skewed toward problems, so they must not feed the rate.
A sampling plan for a mid-sized team could look like this (an illustration, with numbers chosen for the example):
| Stream | What you pull | Per week | Used for |
|---|---|---|---|
| Random baseline | Resolved Fin conversations, random pick | 60 | Correct-resolution rate |
| Risky topics | Refunds, pricing, cancellations, account and data changes | 25 | Policy and money errors |
| Escalations | Conversations Fin handed to a person | 20 | Handover timing and context |
| Bad signals | Negative CSAT, low CX Score, reopened conversations | 15 | Root causes |
| Total | 120 |
Worked example (illustration). Fin is involved in 6,000 conversations a week and the dashboard shows a 65% resolution rate, so 3,900 resolutions. Your reviewers find that 9 of the 60 random resolved conversations were wrong or incomplete: 15%. The corrected rate is 65% x 0.85 = about 55%. With 60 conversations the margin on that 15% is about plus or minus 9 points (1.96 x the square root of 0.15 x 0.85 / 60), so treat one week as a signal and the four-week trend as the number. At about 5 minutes per review, 120 reviews take 10 hours a week, or 5 hours each for two reviewers.
For a tighter margin, raise the random sample, not the targeted streams. Our guide to QA sampling statistics shows how sample size changes the margin.
Intercom’s Monitors can do the pulling for you on the Pro add-on: Intercom describes them as reviewing “a random sample for baseline quality” or “a targeted set based on higher-risk or higher-impact signals”. Without Pro, build the same streams with saved Inbox filters and a spreadsheet.
What should you check in a Fin conversation?
Check whether Fin’s answer matched your source content and your policy, whether the resolution was real, whether any procedure finished, whether the handover came at the right time with context, and whether Fin looped or broke your tone rules. Write each check as a pass rule a reviewer can apply in under a minute, with a named place to find the evidence.
The Fin QA checklist
Copy this into your scorecard or spreadsheet. The “where” column assumes Fin runs in Intercom’s own Inbox.
| # | Criterion | Where to find the evidence in Intercom | Pass rule |
|---|---|---|---|
| 1 | Answer accurate against the source | “Improve answer” on Fin’s reply: the content items Fin found relevant | Every fact in the reply matches a current item in your content. Any wrong fact on money, eligibility or data is a critical fail. |
| 2 | Right source used | The same debugger list, plus the content’s audience setting | The best item for this customer’s case is in the list, and it is targeted at this customer’s audience. |
| 3 | Answered the question asked | The transcript | Fin answered the customer’s actual question, not a similar one, or asked one clarifying question first. |
| 4 | Resolution was real | Conversation status, plus a search for the same customer in the next 7 days | Confirmed by the customer, or assumed with no new conversation or contact on the same issue within 7 days. |
| 5 | Procedure completed | The procedure steps in the conversation, and the backend record (order, refund, account) | The action the procedure promised exists in your system. A promise with no record is a critical fail. |
| 6 | Escalated on the right trigger | Escalation rules and guidance, and the point of handover in the transcript | Fin handed over when your rules say it should, and before the customer asked twice. |
| 7 | Handover with context | The handover message and the teammate’s first reply | The teammate did not have to ask for anything the customer already said. |
| 8 | No loops or repetition | The transcript | Fin did not repeat or rephrase the same answer more than once. |
| 9 | Policy respected | Your policy pages and the reply | No promise outside your refund, fee, delivery or eligibility rules. Critical fail. |
| 10 | Tone and guidance followed | Your communication style guidance and the reply | Tone fits the customer’s state, and terms match your guidance. |
Mark criteria 1, 5 and 9 as critical, so a single failure fails the whole conversation. Intercom’s Scorecards support this directly: a criterion marked Critical sets the whole review to 0% when it fails.
Criterion 8 deserves its own line because Fin teams report it. Intercom says Fin escalates by default when a customer is “stuck in a repetitive loop”, yet in a thread on Intercom’s community one admin wrote that “Something is going wrong with my anti-repetition guidance.” That is one team’s report, but it is the kind of failure only a transcript review shows.
For the handover criteria (6 and 7), our guide to chatbot to human handoff QA has the metrics and a scorecard for both the bot’s decision and the person’s pickup.
How do you score Fin conversations against your scorecard?
Score Fin conversations with the same scorecard structure you use for people, plus the Fin-specific checks above, and record the evidence for every failed criterion. You can do it in Intercom’s Scorecards on the Pro add-on, in a spreadsheet, or in a QA tool connected to the helpdesk where the conversations live. Whatever you use, test the scoring against your own reviewers first.
Three ways to run it:
- Intercom Monitors and Scorecards (Pro add-on). You choose per criterion whether AI or a person scores it. With auto-review on, Intercom marks a conversation as passed when every AI-scored criterion is met and sends failures and edge cases to a person. Intercom’s help page lists calibration workflows as planned, not available yet, so calibrate by hand.
- A spreadsheet. One row per conversation, one column per checklist item, plus cause and fix. Fine up to a few hundred reviews a week.
- A QA tool in your helpdesk. Fin also runs on other helpdesks: fin.ai says it works with Salesforce, Zendesk and others. If your team works in Zendesk or Salesforce, a QA tool connected there can score those conversations.
Before you trust any AI score, have two reviewers score the same 30 to 50 conversations, settle disagreements, then compare the AI’s verdicts with theirs criterion by criterion. Our guide to validating AI QA scoring walks through that test.
How do you feed QA findings back into Fin?
Turn each failed criterion into one change in the part of Fin that caused it: the content, the audience targeting, the guidance, a procedure or an escalation rule. Group failures by cause before fixing anything, test each change in a preview or eval before it goes live, and re-score the affected topic the following week.
| What failed | Likely cause | Where to fix it in Fin |
|---|---|---|
| Wrong or outdated fact | The article, snippet or synced page is wrong or old | Edit it from “Improve answer”, or add a snippet. External content cannot be edited there; fix it at the source, and note that website syncs run weekly. |
| Right article, wrong customer | Content not targeted to the right audience | Set the audience on the content so Fin only uses it for those customers |
| Asked the wrong follow-up, or none | Missing context and clarification guidance | Add one focused guidance rule for that question type |
| Promised an action that did not happen | Procedure step or data connector failed silently | Add a condition after the data connector call that handles errors and empty responses |
| Escalated too late, or not at all | Escalation rules or guidance too narrow | Add a rule on a data attribute, or narrow guidance for the topic |
| Escalated too often | Escalation guidance too broad | Rewrite it for one specific situation |
| Loops and repeated answers | Overlapping guidance rules | Remove rules that conflict; keep each rule to one objective |
Then test before you publish. Intercom’s comparison of its testing tools lists four:
- Previews: a manual test panel across Fin’s training areas, with no limits.
- Batch tests: up to 50 questions at once, rated by a person; the ratings do not train Fin. New workspaces from November 5, 2026 get Evals instead.
- Evals: automated end-to-end tests of Fin’s whole configuration, up to 50 simulations per eval, from a monthly allowance.
- Procedure Simulations: automated tests of one procedure, with no customer-facing output.
One warning from Intercom’s procedure docs: previewing a procedure that is already live can send messages to real customers, so use a simulation for live procedures. Keep your failed conversations as test cases: every fix should pass the conversation that exposed it.
What does a weekly Fin QA routine look like?
A weekly Fin QA routine has five fixed slots: pull the sample on Monday, score it by Wednesday, group failures by cause on Thursday, ship and test fixes on Friday, and check last week’s fixes against this week’s scores. Name one owner for the content and one for guidance and procedures, or fixes stall between teams.
| Day | Task | Owner | Time |
|---|---|---|---|
| Monday | Pull the four streams; check the dashboard for shifts in resolution, involvement and CSAT | QA lead | 30 min |
| Monday to Wednesday | Score the sample against the checklist; record evidence for each fail | 2 reviewers | 5 hours each |
| Thursday | Group fails by cause; pick the top 3 causes by count and cost | QA lead and Fin owner | 45 min |
| Friday | Ship the fixes; test each in a preview, eval or simulation | Content owner, Fin owner | 2 hours |
| Every week | Compare scores on last week’s fixed topics with the week before | QA lead | 20 min |
| Monthly | Report the corrected resolution rate next to Fin’s, with the 4-week trend | QA lead to Head of CX | 1 hour |
Where does Kaizo fit?
Kaizo is QA and governance oversight for support teams where people and AI agents share the work. Its live integrations are Zendesk and Salesforce: AutoPilot scores every resolved conversation in those helpdesks against your own scorecard and attaches a comment that explains each score. If Fin runs on top of one of them, check which Fin conversations reach your helpdesk records before you plan a QA process around them.
See it on your own conversations. Kaizo scores 100% of your support conversations in Zendesk and Salesforce and turns the findings into coaching. Book a demo.
When this is not for you
A formal Fin QA program is overkill if Fin handles a few dozen conversations a week: read them all yourself, with the checklist, in an hour. It is also too early if your help center is out of date across the board. Fix the content first, or every review will report the same root cause. If Fin only answers low-risk questions such as opening hours, a monthly spot check of 20 conversations is enough.
How to start this week
- List your five riskiest Fin topics and the policy page behind each.
- Copy the checklist above into a spreadsheet or an Intercom scorecard, and mark criteria 1, 5 and 9 as critical.
- Pull 30 random resolved Fin conversations and 20 from your riskiest topics.
- Score them with two reviewers and compare where you disagree.
- Fix the most common cause you find, test it in a preview or eval, and re-score that topic next week.
Frequently asked questions
Does Intercom have its own QA for Fin conversations?
Yes, on the Pro add-on. Monitors select which conversations to review, Scorecards define how they are scored, and each criterion can be scored by AI or by a person. Intercom’s help page lists calibration workflows as planned, so test the AI scores against your own reviewers.
Is Fin’s resolution rate a reliable quality metric?
No. It counts conversations where the customer confirmed the answer helped or left without asking for more help, so a customer who gave up counts as resolved. Report it next to a corrected rate from a random sample of resolved conversations you have checked.
Do batch test ratings train Fin?
No. Intercom says batch test ratings are for human review and root cause analysis only. Workspaces created from November 5, 2026 do not get batch tests and use Evals instead.
Can the CX Score replace a QA review?
Not on its own. Its eligibility rules center on conversations with at least two customer replies and two agent or chatbot replies, so many short Fin conversations are not rated. It also rates how the customer felt, not whether the answer matched your policy.
Who is rated when Fin and a teammate both handle a conversation?
It depends on where the CSAT step sits in your workflow. Intercom says a survey placed after the escalation point is sent whether Fin, a teammate or both were involved. On Advanced and Expert plans, the rating data shows which type of agent was rated.
