The metrics that tell you an in-app agent works

Most agent dashboards are built the wrong way round. They start from what the platform emits and arrange it prettily, which produces a page full of numbers that all move and none of which decide anything.

Start from the decisions instead. There are three: ship this change or revert it, expose this new write action or hold it back, and keep investing in this feature or stop, given everything the feature is made of. Every metric below earns its place by informing one of those, and the ones that inform none are trends you read rather than numbers you act on.

This post is the metric set, not the method#

There is a sibling to this one and the division matters. This post is about which numbers to keep and what each one hides. The method for producing the reliability number is a separate piece: how to pick the tasks, how to assert on backend state rather than on the transcript, and how to keep the suite out of the flaky-test graveyard. Read that one when you are building the harness. Read this one when you are deciding what goes on the page everyone looks at.

Task completion, by task#

The share of sessions that reached the user’s actual goal is the only metric here that survives contact with a product decision. Everything else is either a component of it or an explanation for it.

Two rules keep it honest. Define the goal in terms of your own backend rather than the conversation, so “the order exists afterwards” instead of “the agent said it placed the order”. And report it per task, because an average across tasks conceals the thing you need: which task is broken. Three failures spread across three tasks and three failures on one task are different problems with different fixes.

The awkward part is that you have to enumerate the tasks. Most teams discover they cannot, which is itself the finding, and the honest response is to derive the list from real traffic rather than from the roadmap.

Reliability, which is not accuracy#

An agent that works 80 percent of the time fails a daily user roughly once a week, and the failures cluster on the hardest tasks rather than spreading politely.

τ-bench introduced the metric that captures this. Run each task k times and count only the cases where all k attempts succeeded. Its authors reported that agents “are quite inconsistent (pass^8 <25% in retail)” for a mid-2024 model whose single-attempt success was already below 50 percent (Yao et al., 17 June 2024). Date that figure whenever you use it, because models have improved. The gap between usual and always has not closed, since it comes from the variance in tool-using systems rather than from any one model.

The Berkeley Function Calling Leaderboard supplies the other half, which is what to assert on. Its multi-turn evaluation compares “the backend system’s state (excluding the private attributes) after all function calls are executed at the end of each turn”, across 1,000 multi-turn entries in categories including Missing Parameters, Missing Functions and Long-Context (BFCL V3, 19 September 2024, updated 10 December 2024). Whether the order exists is a fact. Whether the sentence about it was well phrased is an opinion, and opinions do not gate releases.

Containment is the metric that flatters you#

Containment counts the sessions that did not reach a human. It goes up when the agent is good and it also goes up when the agent is so bad that people give up, and nothing in the number distinguishes those two.

The industry evidence on this is unusually blunt. Gartner surveyed 5,728 customers and found that only 14 percent of their service issues were fully resolved in a company’s self-service channel, and that even for issues customers described as “very simple”, only 36 percent were handled there (reported 19 August 2024). A later Gartner survey of 265 executives found that “nearly 9 in 10 customer service journeys beginning in self-service are ultimately resolved through multiple channels” (CX Dive, 3 September 2025).

Read those two together and containment stops being a success measure. A journey that leaves your app and lands on the phone line is contained by your dashboard and unresolved by the customer.

Keep containment if your finance team needs it. Put completion next to it, always, and treat a rising gap between them as the alarm rather than as noise. The general argument against optimising the deflection number is set out in why deflection is the wrong thing to measure in support.

Latency, split by owner#

Time from end of speech to the first audible response is the number users feel. Reporting it as one figure hides the only actionable thing about it, which is whose server the wait belongs to.

In our own production measurements a warm turn lands at roughly 2.5 to 3 seconds, and the dominant component is the tenant’s backend rather than the model or our code. A first turn after a cold start is considerably worse, because a connection has to be opened and the prompt cache is empty. Those are different problems: one is fixed by prewarming at app launch, the other by a conversation with whoever owns the slow endpoint. A single average tells you neither. Our measured breakdown of where the seconds actually go has the split.

The retired claims are worth naming since they still circulate. Sub-one-second end-to-end turns and 95 percent accuracy figures are not things we measure or promise, and any target you adopt should come from your own traces.

The three that explain rather than decide#

Repeat use tells you whether the first turn was worth having. Track it at seven and twenty-eight days and expect novelty to inflate the early number. A flat or rising curve after the novelty decays is the closest thing to a verdict on the feature.

Abandon point is more useful than abandon rate. Recording that 6 percent of sessions ended early tells you nothing you can act on. Recording that they ended while the agent was thinking, or immediately after a confirm card appeared, points at a specific fix.

Cost per turn keeps the other numbers honest. Anthropic recommends collecting “the total runtime of individual tool calls and tasks, the total number of tool calls, the total token consumption, and tool errors” for tool-using agents (Writing tools for agents, 11 September 2025). A change that raises reliability and triples the token bill is a change you want to see as a trade rather than as a win.

Two safety numbers that belong on the same page#

Neither of these is a performance metric and both should be able to stop a release.

The count of writes that executed without an approved confirmation must be zero. Not low. If the number is ever non-zero, something is reaching around the gate, and the confirm step as a branch in server code explains why it has to be structural rather than a prompt instruction.

The count of refusals that stayed refusals matters almost as much. Requests the agent should decline are the least-tested path in most deployments, and a regression there is silent because nobody complains when something they should not have been able to do quietly starts working.

What to do in the first month#

Instrument before you launch. The events are cheap and retrofitting them onto traffic you have already lost is not possible. The specific reading of the first weeks of real usage is a different exercise from steady-state measurement, and it is where your task list comes from.

Then pick your eight tasks from that traffic, run each eight times against a resettable backend, and publish the clean-sweep count next to the completion rate. Sixty-four executions is a coffee break, and it is the difference between a dashboard that describes the feature and one that tells you whether to ship.

Common questions#

What is the single most important metric for an in-app agent? Task completion measured per task against backend state. Everything else on a dashboard either explains that number or hides it.

Why is containment a bad primary metric? Because a user who gives up counts as contained. Gartner found only 14 percent of service issues fully resolved in self-service, and nearly nine in ten self-service journeys eventually resolved across multiple channels. Containment cannot see either group.

What is pass^k and why use it here? It is the share of tasks where every one of k attempts succeeded. It answers whether a user can rely on a request, which is what people experience, rather than whether the system usually works, which is what an average reports.

What latency target should I set? Yours, from your own traces, split by which system owned the wait. Our warm turns land around 2.5 to 3 seconds with the tenant backend dominating. Adopting somebody else’s target hides the component you could actually change.

How many metrics should a dashboard carry? Four that can fail a release and four read as trends. More than that and nobody reads the page, which is the same as having none.

Sources#

• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, 17 June 2024. Accessed 12 September 2026.
• Berkeley Function Calling Leaderboard, BFCL V3: Multi-Turn and Multi-Step Function Calling, 19 September 2024, updated 10 December 2024. Accessed 12 September 2026.
• CX Today, Only 1 in 7 Customer Service Queries Resolved With Self-Service, Gartner Study Finds, 19 August 2024, n=5,728. Accessed 12 September 2026.
• CX Dive, CX leaders say self-service and live chat will overtake phone and email, 3 September 2025, n=265 executives. Accessed 12 September 2026.
• Anthropic, Writing tools for agents, 11 September 2025. Accessed 12 September 2026.