An AI agent RFP should test whether the agent can be trusted at scale, not only what it can do. Add questions on how “resolved” is defined and billed, access to full transcripts and action logs, export for independent QA, regression testing, approval controls, audit logs, incidents, model change notice, data terms and EU AI Act support.
In short
- Vendor RFP templates are strong on features, channels and integrations, and weak on how you will check the agent’s work after go-live.
- The question bank below has 38 questions in 8 sections, each with what a good answer looks like and a red flag that should lower the score or end the conversation.
- Treat the billing definition of “resolved”, transcript and log access, and the right to export conversations for your own QA as gates, not as scored features.
- A worked example shows how an 18% audited failure rate inside billed resolutions turns a $1.00 resolution into $1.22 per correct resolution.
- Close with a scripted demo on your own conversations: eight tasks, the same order for every finalist, scored on the same scale as the written answers.
This article is for teams buying an AI agent platform for customer support, whether the shortlist includes Sierra, Decagon, Intercom Fin, Salesforce Agentforce, Zendesk AI agents, Ada or an in-house build. It does not rank vendors. It gives you the questions their own templates leave out. If you are buying QA software rather than an AI agent, use our QA software RFP with 40 questions and a scoring sheet instead; this page follows the same format.
What should an AI agent RFP include beyond features?
An AI agent RFP should include, beyond features, the terms that decide whether you can check and control the agent: how a billed resolution is defined, what evidence you can see and export, how changes are tested and announced, which actions need approval, how incidents are handled, and where your data goes. Features decide the demo; these decide the second year.
Most published templates, many written by AI agent vendors, cover channels, languages, integrations, knowledge sources, handoff, security and price well, and go less deep on the questions buyers raise once the agent is live. The table compares a typical template (several published on vendor sites, read on November 13, 2026) with this question bank.
| Topic | Typical vendor template | This question bank |
|---|---|---|
| Channels, integrations, knowledge sources | Covered in depth | Left to your feature section |
| How “resolved” is defined | Often asked | Asked, with the trigger, the time window and the verification |
| Disputing billed resolutions | Rarely, and without a process | A dispute window, evidence rules and credits |
| Full transcripts and action logs | Partly (summaries, some logs) | Raw transcripts, tool calls and the knowledge used, per conversation |
| Export for independent QA during the term | Mostly at contract exit only | Continuous export, in a documented format |
| Regression testing after go-live | One test before signing | A test suite rerun before every change |
| Model change notice | Sometimes, not scored | Written notice, a test window and a fallback |
| AI-specific incident process | Rarely | Severity, notice time, stop control and a written report |
| EU AI Act deployer support | Not found | Roles, disclosure, documentation and logs |
The gap is not an accident: a sales team can answer a feature question in a demo, but billing disputes or model change notice need a written answer from legal and product that becomes a contract term.
Why does the definition of “resolved” belong in the RFP?
The definition of “resolved” belongs in the RFP because it is usually both the main success metric and the billing unit, and vendors define it differently. A resolution counted when the customer simply stops replying is not the same as one checked against the answer, the action and repeat contact. The difference shows up on your invoice.
Two public examples show the range. Intercom counts a Fin resolution when, after Fin’s last answer, the customer confirms it helped or “exits the conversation without requesting further assistance”, and deducts the charge if the customer comes back to that conversation for more help (Intercom help center). Zendesk bills on automated resolutions and, for its verified tier, has a large language model review the conversation once it ends, which for messaging is 2 hours after the first message by default and for email 72 hours (Zendesk help center).
Neither definition, as documented, checks whether the answer was correct against your policy or whether the promised refund was processed. That is your job, and your RFP should ask how the vendor helps you do it. Our guide to auditing an AI agent resolution rate gives the sampling method, and correct resolution rate is the metric to report next to the vendor’s number.
Worked example: what an audit does to the price of a resolution
This is an illustration with made-up numbers, not data from any vendor.
| Line | Value |
|---|---|
| Conversations handled by the AI agent in a month | 40,000 |
| Billed resolutions reported by the vendor | 24,000 (60%) |
| Price per resolution (illustrative) | $1.00 |
| Monthly invoice | $24,000 |
| Billed resolutions audited (random sample) | 400 |
| Failed the audit (wrong answer, action not done, policy breach, or repeat contact within 7 days) | 72 (18%) |
| Estimated correct resolutions | 24,000 × 0.82 = 19,680 |
| Cost per correct resolution | $24,000 ÷ 19,680 = $1.22 |
| Margin of error on 18% with 400 audited (95%) | about ±3.8 points |
| Credit if failed resolutions are refunded | 24,000 × 0.18 × $1.00 = $4,320 a month |
The margin of error comes from 1.96 × √(0.18 × 0.82 ÷ 400) = 0.038. So the true failure rate is likely between about 14% and 22%, which is precise enough to negotiate on. Without a dispute clause in the contract (questions H2 and H3 below), the $4,320 stays on the invoice every month.
The AI agent RFP question bank (38 questions to copy)
These 38 questions cover the eight areas where AI agent platforms differ most on quality and governance. Copy them next to your feature questions, keep the numbering, and ask for evidence with each answer: a screenshot, a document, a contract clause or a live demonstration. A “yes” without evidence scores as no answer.
Section A: Resolution definition and billing (5 questions)
| # | Question | What a good answer looks like | Red flag |
|---|---|---|---|
| A1 | What exactly triggers a billed resolution, per channel? | A written rule per channel: the trigger, the time window and what is excluded | “Our AI decides”, with no written rule |
| A2 | How is a resolution verified before it is billed? | A described check (customer confirmation, LLM review, repeat-contact window) you can see per conversation | Silence from the customer counted as success, with no window |
| A3 | What happens when the customer comes back about the same issue? | The resolution is reversed or not billed within a stated window, such as 7 days | Repeat contact billed as a new resolution |
| A4 | Can we see, per billed conversation, why it counted as resolved? | A field or log entry per conversation, exportable | Only a monthly total on the invoice |
| A5 | Do you report resolutions separately from containment and deflection? | Separate definitions and separate numbers | One “automation rate” that mixes them |
Section B: Transcripts, action logs and export (5 questions)
| # | Question | What a good answer looks like | Red flag |
|---|---|---|---|
| B1 | Can we access the full transcript of every AI agent conversation, including internal steps? | Every message, plus the knowledge articles and instructions used for each answer | Summaries only, or a sample |
| B2 | Do you log every action and tool call the agent makes, with inputs and results? | A per-conversation log: which system, which call, which parameters, success or failure | “Actions are logged in the target system” with nothing on your side |
| B3 | Can we export all conversations and logs continuously during the contract, for our own QA? | An API or scheduled export in a documented format, at no extra cost, to a tool of your choice | Export only at termination, or only to the vendor’s own analytics |
| B4 | How long do you keep transcripts and logs, and can we set the period? | A default retention period and a configurable one, per data type | No stated retention, or deletion you cannot control |
| B5 | Can an independent reviewer see the same evidence your dashboards use? | The same conversation, steps and outcome fields, exportable | A score or label with no access to the underlying conversation |
Question B3 is the one most templates skip and the one that matters most if you want to score AI agent conversations on your own scorecard, as we describe in our guide to AI agent evaluation.
Section C: Testing and regression (5 questions)
| # | Question | What a good answer looks like | Red flag |
|---|---|---|---|
| C1 | Will you run the agent on a set of our historical conversations before we sign, and share every result? | A test on several hundred of your real conversations, with every answer visible | A demo on the vendor’s own sample data only |
| C2 | Can we build and save our own test cases, including expected answers and expected actions? | A test library you own and can export | Tests only the vendor’s team can create |
| C3 | Can we rerun the full test set before every change to prompts, knowledge, policies or the model? | One-click or scheduled regression runs, with a comparison to the last run | Testing is a one-off at onboarding |
| C4 | Can we simulate difficult customers: off-topic questions, angry customers, attempts to get a refund outside policy? | Simulated conversations from scripts or personas, with results you can review | “The model handles that” with no way to check |
| C5 | Can we block a release when tests fail? | A release gate with a pass threshold you set | Changes go live as soon as they are saved |
Our guide to AI agent testing explains how to build a golden test set from real conversations and what a release gate should check.
Section D: Policy and approval controls (5 questions)
| # | Question | What a good answer looks like | Red flag |
|---|---|---|---|
| D1 | Which actions can the agent take alone, and can we require approval per action type? | A list of actions with a setting per action: allowed, needs approval, blocked | All actions on or all off |
| D2 | How are limits enforced, for example a refund ceiling? | Limits checked by the system before the action runs, not only stated in the prompt | Limits written only as instructions to the model |
| D3 | Who can change prompts, policies and knowledge, and is every change approved and versioned? | Roles, an approval step, version history and rollback | Any admin edits live, with no history |
| D4 | Can we limit the agent to specific topics, customers or channels? | Scoping by intent, segment and channel | One global switch |
| D5 | Can we stop the agent quickly, for one topic or entirely, and route to people? | A documented stop control per topic and globally, with a fallback route | Stopping requires a vendor ticket |
Section E: Handoff and escalation (4 questions)
| # | Question | What a good answer looks like | Red flag |
|---|---|---|---|
| E1 | Which triggers send a conversation to a person, and can we change them? | Configurable triggers: request for a human, frustration, risk topics, repeated failure | A fixed set you cannot see |
| E2 | Can a customer always reach a person, and how fast? | A stated path and time per channel | The agent keeps the customer when they ask for a human |
| E3 | What does the human agent receive at handoff? | The full transcript, the actions taken and the reason for the handoff | A one-line summary |
| E4 | How do you report handoffs that came too late? | A report of conversations with several failed attempts before handoff | Handoff counted only as a failure of the AI |
An agent that keeps a customer who has asked for a person turns one bad answer into a complaint. See our chatbot to human handoff scorecard for what to score.
Section F: Change management and incidents (5 questions)
| # | Question | What a good answer looks like | Red flag |
|---|---|---|---|
| F1 | Will you notify us before changing the underlying model or its version, and how far ahead? | Written notice a set number of days ahead, with release notes | “We continuously improve the model” |
| F2 | Can we test a model change on our test set before it reaches our customers? | A preview or staging environment for the new model | Changes reach production first |
| F3 | Can we stay on the previous model for a period if our tests fail? | A rollback or pinning option for a stated period | No option to delay |
| F4 | What is your incident process for harmful or wrong answers at scale? | Severity levels, notice times, a named contact, and a written report after each incident | Only an uptime SLA |
| F5 | How do you detect quality drift when nothing in our setup changed? | Monitoring on answer quality, with alerts you can set | Drift is found by your customers |
Drift without a configuration change is a real buyer worry: in one Intercom community thread (November 2025), a customer reported that Fin began giving wrong answers although “All the Content stayed the same” (Intercom community). F1 and F5 exist for that moment; our guide to AI agent monitoring covers the alerts to set.
Section G: Data, privacy and regulation (5 questions)
| # | Question | What a good answer looks like | Red flag |
|---|---|---|---|
| G1 | Where is our data stored and processed, including by the model provider? | Named regions, and a region choice where you need one | “In the cloud” |
| G2 | Who are your sub-processors, including model providers, and how do you notify us of changes? | A published list and advance notice with a right to object | A list available “on request” with no notice |
| G3 | Do you or your model providers use our conversations to train or improve models? | A clear written no, or an opt-in, in the contract | An opt-out buried in the terms |
| G4 | Under the EU AI Act, which role do you take, and how do you support us as deployer? | A stated role, the disclosure that the customer is talking to an AI, documentation, and logs you can keep | “The AI Act does not apply to us” |
| G5 | Can we audit you, or receive independent audit reports? | Current reports under NDA and an audit clause | No reports, or reports for a different product |
Some of these rest on law. Under the GDPR, a processor may not engage another processor without your written authorization, and under a general authorization it must tell you about intended changes so you can object; it must also allow for audits (GDPR Article 28). When the vendor’s model provider is a sub-processor, a switch of provider should reach you through that notice. A new model version from the same provider usually will not, which is why F1 asks for it separately.
The EU AI Act asks providers to make sure people know they are interacting with an AI system, from August 2, 2026 (European Commission FAQ on Article 50). AI literacy duties for providers and deployers have applied since February 2, 2025 (European Commission). If a use counts as high-risk, deployers must also monitor the system and keep its logs for at least six months (Regulation (EU) 2024/1689, Article 26). Most support agents are not high-risk, but your legal team decides that, not the vendor. Our article on the EU AI Act in customer service explains the risk classes.
Section H: Commercial terms and disputes (4 questions)
| # | Question | What a good answer looks like | Red flag |
|---|---|---|---|
| H1 | What do we pay for a conversation that is not resolved, handed off, or abandoned? | A price table per outcome type | Charges for outcomes that are not in the table |
| H2 | How do we dispute a billed resolution, and within what window? | A dispute process with a window, the evidence accepted (for example your audit) and a timeline | Disputes “handled case by case” |
| H3 | Do you credit resolutions that our audit shows were wrong? | Credits, or a price adjustment, when the audited failure rate passes an agreed threshold | Billing based only on the vendor’s own count |
| H4 | What happens to price and volume commitments when resolution quality falls below an agreed level? | A remedy: credits, a right to pause, or a right to exit | Minimum commitments with no quality condition |
Contract terms that let you dispute and credit resolutions make the audit in the worked example pay for itself.
How do you score AI agent RFP answers?
Score AI agent RFP answers in two passes. First run the gates: any vendor that fails one is out, whatever its feature score. Then score every remaining answer from 0 to 3 against written definitions, weight the sections, and break close totals with the scripted demo rather than a longer spreadsheet.
Use the same 0 to 3 scale as in our QA software RFP: 0 for no answer, 1 for major limits or “on the roadmap”, 2 for meets the requirement with partial evidence, 3 for meets it with evidence shown. A “yes” with no evidence scores 0.
| Gate (knockout) | Questions | Why it is a gate |
|---|---|---|
| A written definition of a billed resolution | A1, A2 | You cannot check an invoice without it |
| Full transcripts and action logs per conversation | B1, B2 | No evidence, no QA and no incident review |
| Continuous export for your own QA | B3 | Your quality numbers would depend only on the vendor’s |
| A stop control and an always-available human path | D5, E2 | Customer harm while you wait for a ticket |
| Written data terms: residency, sub-processors, training | G1 to G3 | Legal will block the contract later anyway |
A starting weighting for teams that already have the feature shortlist; agree it with support, IT, legal and procurement before the answers arrive, and do not change it after.
| Section | Weight |
|---|---|
| A. Resolution definition and billing | 15 |
| B. Transcripts, logs and export | 15 |
| C. Testing and regression | 20 |
| D. Policy and approval controls | 15 |
| E. Handoff and escalation | 10 |
| F. Change management and incidents | 15 |
| G. Data, privacy and regulation | 5 (after the gate) |
| H. Commercial terms and disputes | 5 |
| Total | 100 |
The weighted score is the sum of each section’s weight times its average score, divided by 3, so a vendor that scores 3 everywhere gets 100. Have two people score separately and discuss every question where they are more than one point apart. Our AI agent governance controls help you decide which of sections D and F your organization needs most.
What should a scripted AI agent demo include?
A scripted AI agent demo should show each finalist running your own conversations through the same tasks, in the same order, within the same time. Send the script a week ahead, ask for your test set to be loaded, and score each task on the 0 to 3 scale. You are testing the product’s controls, not the presenter.
Plan 90 minutes per vendor. Bring 30 to 50 real conversations from your queue, with personal data removed, including ones your team handled badly.
| Step | Task for the vendor | What you check | Linked questions |
|---|---|---|---|
| 1 (10 min) | Answer 10 of your conversations live, chosen by you on the day | Right answer, right tone, sources shown | C1 |
| 2 (10 min) | Open one billed resolution and show why it counted | The trigger, the window and the verification for that conversation | A1, A2, A4 |
| 3 (10 min) | Show the full transcript and action log of a conversation that issued a refund or changed an order | Each tool call, its parameters and its result | B1, B2 |
| 4 (10 min) | Export last week’s conversations in the format you would receive | Fields, format, completeness | B3 |
| 5 (15 min) | Change one policy (for example, lower the refund ceiling), rerun the test set, compare results | Versioning, approval, regression report, release gate | C3, C5, D2, D3 |
| 6 (10 min) | Run three hostile scenarios: an off-topic request, a refund outside policy, a customer asking for a human | Refusals, limits held, handoff with context | C4, E1, E3 |
| 7 (10 min) | Stop the agent for one topic and show where the conversations go | Time to stop, the fallback route | D5, E2 |
| 8 (15 min) | Walk through the last model change: the notice, the test window, the rollback option | Release notes, staging, pinning | F1 to F3 |
After the demo, ask two references with a similar volume and channel mix three questions: what went wrong in the first three months, how the vendor handled it, and how a model change was announced. Their answers to the last one tell you whether F1 is a policy or a promise.
How does this fit with QA once the agent is live?
The RFP answers become your operating model once the agent is live: the export from B3 feeds your quality reviews, the test set from C2 becomes your regression suite, and the dispute process from H2 runs on your audit results. A vendor that scores well on B, C and H makes independent QA cheap.
In practice, many teams score a sample or all of the AI agent’s conversations on the same scorecard they use for human agents, add checks for invented facts and policy, and report a corrected resolution rate next to the vendor’s. If you want that scoring automated, Kaizo scores 100% of AI agent conversations against your own scorecard, inside the helpdesk where they already live, and links every score to the evidence in the conversation. See QA for AI agents for how it works.
When is this RFP more than you need?
This question bank is more than you need when the AI agent will only answer a narrow set of informational questions, takes no actions in your systems, and handles a few hundred conversations a month. In that case, the gates and sections A, B and E are enough. Add the rest when the agent starts issuing refunds, changing orders or handling regulated topics.
It is also the wrong tool for a first look at the market: run a short request for information or a few discovery calls first, then send this RFP to three to five finalists.
How to start your AI agent RFP this week
Pull 30 to 50 real conversations for the demo and the pre-signing test, write your gates, copy the 38 questions next to your feature questions, fix the weights in one meeting, and send the RFP with a two-week deadline.
- Pick the 30 to 50 conversations, including your five hardest cases.
- Write your gates from the table above, and remove any you do not need.
- Ask legal which EU AI Act and GDPR questions in section G they want answered in writing.
- Ask finance to model your monthly bill under each vendor’s resolution definition, using last month’s volume.
- Book the scripted demos for the week after answers arrive.
If you want to see how scoring every AI agent conversation on your own scorecard would work alongside the platform you choose, book a demo and bring your shortlist.
Frequently asked questions
What is an AI agent RFP?
An AI agent RFP is a request for proposal sent to vendors of AI agents for customer service, so their answers can be compared on the same questions. It describes your volume, channels and systems, lists your requirements and knockouts, and asks each vendor to answer with evidence. A complete one covers features and the quality, governance, data and billing terms that decide whether the agent can run at scale.
How many vendors should get an AI agent RFP?
Three to five vendors is usually enough. Fewer gives little to compare, and more creates scoring and demo work that rarely changes the outcome. Narrow the list with discovery calls first and drop any vendor that fails a gate on that call, for example one that cannot export conversations during the contract.
Should we pay per resolution or per conversation?
Per resolution aligns the vendor’s price with outcomes, but only if “resolved” is defined in writing, checked, and open to dispute. Per conversation is simpler to predict and audit. Whichever unit you choose, model last month’s volume under each vendor’s definition and ask for credits when your audit finds billed resolutions that were wrong.
How long should an AI agent pilot run before signing?
A pilot should run long enough to see repeat contact and a few policy changes, which usually means four to eight weeks on a defined set of topics. Agree the success criteria, the audit method and the exit terms before it starts, and run your test set before and after any change the vendor makes during the pilot.
Who should be involved in evaluating an AI agent vendor?
Support operations owns the requirements and the demo. QA owns the test set and the scoring of conversations, IT owns integrations and actions, legal and privacy own section G and the contract, and finance owns the billing model. Name one decision owner before the RFP goes out so the weights are fixed in advance.
