AI agent observability is the practice of tracing what an AI agent does, step by step, so engineers can see latency, token costs, tool errors and failed runs. For a support leader that is half the job. The other half is quality monitoring: checking that the agent gave customers the right answer, followed policy, escalated when it should and actually did what it said it did.
In short
- Technical observability answers “is the agent running?”. Quality monitoring answers “is the agent right?”. Observability tools are built mainly for the first.
- The costliest support failures look like success in the logs: wrong answers on resolved conversations, missed escalations, policy breaches and actions the agent claimed but never completed.
- A useful AI agent monitoring dashboard gives each metric a definition, a data source, an alert threshold and one owner. The spec below has 13 rows to copy.
- Set thresholds from your own 4 week baseline and read every metric against a change log, so drift after a prompt or knowledge update shows up within days.
- A 45 minute weekly review that ends with named fixes keeps the dashboard from becoming wallpaper.
What is AI agent observability?
AI agent observability means recording every run of an AI agent as a trace: the customer’s message, each model request, each tool or API request, the data retrieved and the reply, with timings, token counts and errors. Engineers use those traces to debug failures, control cost and keep the agent fast and available.
The vocabulary comes from software operations. Google’s Site Reliability Engineering book names the four golden signals of monitoring: latency, traffic, errors and saturation. AI agent observability adds what is specific to language models: prompts, model versions, token usage, retrieval and tool use. The industry is converging on a shared data format, the OpenTelemetry semantic conventions for generative AI.
A trace is the record of one agent run, and each step inside it (a model request, a knowledge search, an order lookup) is a span. Demand for this visibility is real: in a 2025 IBM Institute for Business Value study, roughly half of leaders named “a lack of visibility into agent decision-making processes as a significant implementation barrier” (IBM).
What is the difference between AI agent observability and AI agent monitoring for quality?
Technical observability tells you whether the AI agent ran: it answered quickly, without errors, at an acceptable cost. Quality monitoring tells you whether the agent was right: correct answer, allowed action, right escalation, customer better off. A conversation can be perfect in the trace and wrong for the customer, and the trace will not show it.
LangChain, which builds the LangSmith observability platform, puts it plainly: traditional monitoring tools “focus on metrics like latency, traffic, errors, and saturation”, while “agent observability requires monitoring the inputs and outputs themselves, not just the system metrics around them” (LangChain, February 2026).
The legal exposure sits on the quality side. In Moffatt v. Air Canada (2024 BCCRT 149), the airline’s website chatbot told a customer he could claim a bereavement fare retroactively, which policy did not allow. The tribunal ordered the airline to pay and wrote that “it should be obvious to Air Canada that it is responsible for all the information on its website” (tribunal decision). No latency chart would have flagged that answer.
| Technical observability | Quality monitoring | |
|---|---|---|
| Question | Is the agent running, fast and affordable? | Is it giving right answers and taking right actions? |
| Unit of data | Trace and span | Conversation, scored against a scorecard |
| Typical metrics | Latency, error rate, tool failures, tokens, cost | Accuracy, policy, escalation, handoff, sentiment, repeat contact |
| What it misses | A confident wrong answer that closed the ticket | A broken integration, unless it surfaces in the conversation |
| Usual owner | AI engineering, platform team | Support operations, QA lead, AI agent owner |
The layers meet in the middle: a loop shows up in the trace as repeated steps and in the conversation as a frustrated customer. Scoring the conversations themselves against a written standard is what support teams call AI agent QA.
What should a support leader monitor in an AI agent?
A support leader should monitor seven things: answers that were wrong but counted as resolved, escalation and handoff failures, policy breaches, actions claimed but not completed, loops, sentiment drops, and drift after each change. Every one of them passes a technical health check, which is why each needs its own measure.
- Wrong but resolved. The customer stopped replying, so the helpdesk counted a resolution, but the answer was wrong. This is the core of the silent failures your dashboard counts as wins, and hallucinations are one cause. Only scoring the content of resolved conversations reveals it.
- Escalation and handoff failures. A missed escalation is a case the agent kept when your rules say a person should take it. A failed handoff is an escalation done badly: no context passed, the customer repeats everything, or nobody picks it up in time.
- Policy breaches. The agent promised a refund, fee waiver or exception your policy does not allow. One such promise can be enough for leadership to pause every further AI rollout until compliance has reviewed it.
- Claimed but not done. The agent says “your refund is on its way”, but the backend shows no refund. The tool request failed silently, or the agent described an action it never attempted. Catching it means reconciling transcripts with the system of record.
- Loops. The agent asks the same question three times or bounces the customer between chat and phone. The trace shows a high turn count; the customer feels stuck and comes back, or cancels.
- Sentiment drops. The customer ends angrier than they started, even on a technically resolved case. A rise on one topic usually points at one answer the agent handles badly.
- Drift after changes. Quality moves when something ships: a model version, a prompt edit, a rewritten article, a new policy or integration. Read every metric against a change log and drift has a date and a cause.
Whether the agent is worth running at all (resolution, containment, cost per resolution, CSAT) is a separate question, covered in how to measure AI agent performance. Monitoring is the layer underneath: catching what went wrong this week, while it is still small.
The AI agent monitoring dashboard spec
A monitoring dashboard works only if each metric has a precise definition, a named data source, an alert threshold and a single owner. Without those four fields a chart turns red and nobody moves. Copy the spec below and replace the example thresholds, which are illustrations, not benchmarks, with numbers from your own baseline.
| Metric | Definition | Data source | Alert threshold (example) | Owner |
|---|---|---|---|---|
| Wrong but resolved rate | Share of AI-resolved conversations failing the accuracy criterion | QA scoring of resolved conversations | 0.5 points above the 4 week baseline, or above 3% | AI agent owner |
| Policy breaches | Conversations where the agent promised or granted something outside written policy | Scorecard policy criterion | Any single case on a red-list topic, same day; otherwise above 0.5% weekly | Support operations with compliance |
| Missed escalation rate | Conversations your rules send to a person that the agent kept | Scorecard escalation criterion | Above 1% weekly, or any on a red-list trigger | AI agent owner |
| Handoff failure rate | Escalations with no context passed, a repeating customer, or no pickup within target | Helpdesk handoff events plus scorecard | Above 10% of handoffs, or pickup above service level for 2 days | Support operations lead |
| Claimed but not done | Agent states an action (refund, cancel, change) with no matching backend record | Transcript claims reconciled with order, billing or account logs | Any case on payments; otherwise above 0.2% of action conversations | AI engineering with support operations |
| Loop rate | Same question or answer repeated 3 or more times, or a channel bounce | Traces and helpdesk channel history | Turn count above the baseline 95th percentile on more than 2% of conversations | AI agent owner |
| Repeat contact rate | Same customer, same issue, back within 72 hours of an AI resolution | Helpdesk contact history | 25% relative rise on a topic, week over week | Support operations lead |
| Negative sentiment shift | Share of conversations where sentiment ends lower than it started | Sentiment scoring | 5 point rise on a topic, week over week | CX lead |
| Quality score by topic | Average scorecard result per contact reason | QA scoring | Any topic 5 points below its baseline | QA lead |
| Post-change delta | Each metric for 7 days after a change versus the 7 days before | Change log plus the rows above | Any threshold crossed within 7 days of a change triggers a rollback review | Owner of the change |
| Scoring coverage | Share of AI-handled conversations actually scored | QA tool versus helpdesk volume | Below 95% for a day | QA operations |
| Scorer agreement | Agreement between automated scores and human reviewers on a weekly sample | Calibration sample | Below the level agreed at go-live | QA lead |
| Technical health and cost | Error rate, timeouts, p95 latency, tool failures, cost per conversation | Traces, APM and billing | Your engineering service levels and budget | AI engineering |
Three rules keep it honest. Every quality metric links to the conversations behind it, so a red cell opens real examples instead of a debate. Every metric is cut by topic, because an average hides the one article that is wrong. And the first two rows outrank the last: a fast, cheap agent that gives wrong answers is not healthy.
Most rows depend on a written scorecard. If you do not have one, start with the AI agent scorecard and the methods in the guide to AI agent evaluation.
How do you set alert thresholds for AI agent monitoring?
Set alert thresholds from your own baseline. Score at least 4 weeks of conversations, note the normal weekly range of each metric, then alert on a move outside that range or on an absolute ceiling, whichever comes first. Red-list events such as payment errors or legal promises alert on a single case, whatever the rate.
Google’s SRE guidance is that “every page should be actionable”, so use three tiers:
- Same day: a red-list policy breach, a claimed but not done payment, coverage below 95%, an outage.
- Within 2 days: a rate threshold crossed on a busy topic, or any threshold crossed within 7 days of a change.
- Weekly review: slower trends, low-volume topics, sentiment and repeat contact.
Worked example: why full coverage changes what you can see
An illustration with round numbers, computed for this article. An AI agent handles 50,000 conversations a week and resolves 35,000 without a person. The wrong but resolved baseline is 2.0%, about 700 wrong answers a week. A returns article is rewritten on Monday, and the next week the rate is 2.8%: about 980 wrong answers, 280 more than usual.
| Score a 2% sample | Score every resolved conversation | |
|---|---|---|
| Conversations scored | 700 | 35,000 |
| Wrong answers at baseline (2.0%) | 14 | 700 |
| Wrong answers after the change (2.8%) | about 20 | 980 |
| Normal weekly noise (about the square root of the count) | about 4 | about 26 |
| Is the change visible? | No: 6 extra cases, within noise | Yes: 280 extra cases, over 10 times the noise |
Policy breaches are rarer. At 0.1%, there are about 35 a week in 35,000 resolved conversations. A 2% sample holds 0.7 on average, so the chance it contains none is about 50%. In half the weeks, a sample-based review reports zero breaches while 35 customers were told something your policy does not allow.
What does a weekly AI agent quality review look like?
A weekly AI agent quality review is a 45 minute meeting where the AI agent owner, the QA lead, support operations and the knowledge owner go through alerts, failure clusters and last week’s changes, and leave with named fixes. Compliance joins when a policy breach is on the list. The output is a fix list and an updated change log.
- Alerts since last week (5 minutes): what fired, what was done, what is open.
- Top 3 failure clusters (15 minutes): 3 real conversations each, read on screen, not summarized.
- Changes shipped (10 minutes): each change from the log with its 7 day before and after numbers.
- Decide fixes (10 minutes): one owner and one date per fix, whether a knowledge edit, guardrail, escalation rule or rollback.
- Check last week’s fixes (5 minutes): did the metric each fix targeted move?
Run fixes through your pre-launch checks before they ship; AI agent testing covers test sets that stop a fix for one topic breaking another. Who signs off on red-list changes, and who can switch a topic off, belongs in your AI agent governance model.
What types of AI agent monitoring tools are there?
AI agent monitoring tools fall into five categories: developer tracing tools, application performance monitoring suites, the agent platform’s own analytics, conversation QA tools that score resolved conversations, and AI security tools. Each answers a different question, which is why most support teams with an AI agent in production end up using two or three of them.
| Category | Question it answers | Typical user | Examples | Where it stops |
|---|---|---|---|---|
| Developer tracing | Why did this run fail, and what did it cost? | AI engineers | LangSmith, Langfuse, Arize Phoenix | Built to debug runs, not to judge answers against your policy |
| Application performance monitoring | Is the service up and fast? | Platform teams | Datadog, IBM Instana, Splunk | Thin on answer correctness |
| Agent platform analytics | How is this agent doing on its own measures? | AI agent owner | Built into Fin, Decagon, Agentforce, Zendesk AI agents | One platform, its own definitions |
| Conversation QA | Was the answer right, allowed and well escalated? | QA lead, support operations | QA tools that score conversations on your scorecard | Works after the conversation ends; not a run debugger |
| AI security | Is the agent doing anything unsafe or unauthorized? | Security team | AI security platforms | Access and misuse, not service quality |
Choose by the question you cannot answer today: if engineers cannot debug a failed run, start with tracing; if nobody can say how many resolved conversations were wrong last week, start with conversation QA.
When is this level of AI agent monitoring not for you?
Full quality monitoring is not worth building while the AI agent handles a few hundred conversations a week on low-risk topics. At that volume one person reading conversations weekly is enough, and the time is better spent on the knowledge base.
It is also the wrong first step if the agent is not live yet (you need pre-launch testing) or if you have no written standard for a good answer: a tool cannot alert on a rule nobody has written down.
How to start AI agent monitoring this week
You can start AI agent monitoring this week without buying anything: write a red list, start a change log, read last week’s resolved conversations for a first baseline, and name an owner for every metric. Tooling can come once you know which questions you cannot answer.
- Write the red list: 5 to 10 topics such as refunds, cancellations, legal, safety and vulnerable customers.
- Start a change log: date, what changed, who changed it. Backfill the last month.
- Read 200 resolved AI conversations from last week against your policy, and count wrong but resolved, missed escalations and breaches.
- Copy the dashboard spec, delete rows you cannot source yet, and fill every owner cell.
- Book the weekly review in the same slot every week.
If your conversations live in Zendesk or Salesforce and you want the quality layer without reading by hand, that is the part Kaizo covers. AutoPilot scores every conversation against your own scorecard when it resolves and links each score to the evidence, and you test it against your past ratings before it goes live. Root-cause analysis traces a quality dip back to the conversations that produced it. It does not replace your engineers’ tracing tools; see QA for AI agents for how it fits.
See it on your own AI agent’s conversations. Book a demo.
Frequently asked questions
Can AI agents be monitored?
Yes, on two levels. An agent’s runs can be traced for errors, speed and cost, and its conversations can be scored for accuracy, policy and escalation. Support teams need both, because a technically healthy run can still give a customer a wrong answer.
What are the best tools for monitoring agentic AI?
There is no single best tool, because each category answers a different question: tracing tools debug runs, performance suites watch uptime, platform analytics report on one agent, and conversation QA tools check whether answers were right and allowed. Start with the question you cannot answer today.
What is an AI agent trace?
A trace is the full record of one AI agent run, from the customer’s message to the reply. Each step inside it, such as a model request or an order lookup, is a span with its own timing, inputs, outputs and errors. A trace shows how the agent reached an answer, not whether the answer was correct.
Who should own AI agent monitoring in a support team?
Split ownership by layer. AI engineering owns technical health and cost. A named AI agent owner, usually in support operations, owns answer quality, escalation and loops. The QA lead owns the scorecard and scorer agreement. Every metric on the dashboard should carry exactly one name.
Is containment rate enough to monitor an AI agent?
No. Containment counts every conversation that did not reach a person as a success, including those where the agent was wrong and the customer gave up. Read it next to the wrong but resolved rate, repeat contacts within 72 hours and sentiment shift. See QA for AI-handled conversations for what to check first.
