What the first week of real assistant traffic teaches you

You will have spent weeks on the tool surface and the prompt, tested it against a list of questions you wrote yourself, and shipped it to a small percentage of users. Then the logs arrive and the product turns out to be slightly different from the one you built.

This is the good outcome. The week is worth planning for, because most of what it teaches is only legible if you instrumented for it beforehand.

The three things only real traffic tells you#

What people ask, in their own words, which is never the phrasing in your test list. Which of those requests your tools cannot serve, which is the actual roadmap. And why turns fail, which splits into about four causes that need different fixes and look identical in an aggregate success rate.

Everything else you could have guessed.

What people ask, versus what you designed for#

Plot the requests by intent and you get two curves. The one you designed for is short and fat: five or six things, each of them common. The one you observe has those five or six at the front and then a long tail that runs for pages.

Two patterns show up almost every time.

People ask compound questions. Not “what’s my balance” but “what’s my balance and did the Tuesday payout land”. Your tools answer each half and the model has to compose, which it does correctly most of the time and visibly slowly every time. That is a signal to add one tool, not to rewrite the prompt.

And people ask about state you did not think of as state. Was that refund processed. Is my address still the old one. Why is this pending. These are read questions against data you already have, and they are cheap to add once you know they were asked.

The long tail is where the product is#

The instinct with a long tail is to dismiss it as noise. Read a hundred of them instead.

Sort the tail into three piles. Requests a tool already covers but the model did not map to it, which is a description problem. Requests no tool covers but your backend could answer, which is the roadmap. And requests your product genuinely does not do, which are worth counting because a large enough pile is a feature request from people who have already tried to use it.

The third pile is the one nobody logs and the one worth the most.

The turns that failed, sorted by why#

An aggregate failure rate is a number you cannot act on. A taxonomy is.

The Berkeley Function Calling Leaderboard walks through three failure scenarios that map cleanly onto product bugs. “Failure to perform implicit actions”, where the model does not break a task down into the calls it needs. “Failure to understand the current state before performing action”, where even the best models “sometimes fail to explore the current state before performing actions, which can be dangerous if the actions are non-reversible”. And “unnecessary planning and thinking”, where the model plans a step the request never needed (BFCL V3, 19 September 2024, updated 10 December 2024).

Add two of your own: the tool call that returned an error, and the turn where the backend was simply slow. Five buckets is enough to route every failure to the person who can fix it.

Anthropic’s guidance on evaluating tool-using agents says to collect, alongside accuracy, “the total runtime of individual tool calls and tasks, the total number of tool calls, the total token consumption, and tool errors” (Writing tools for agents, 11 September 2025). Those four are the measurements that turn a week of logs into a set of schema changes.

The metric that looks good and means nothing#

Containment. The percentage of conversations that ended without a handoff.

It rises when the assistant works, and it rises just as reliably when the user gives up. Those two outcomes are indistinguishable in a transcript, so a containment number is a measurement of two things added together, one of which you want and one of which you do not.

The independent evidence is unkind to it. Gartner research covering 5,728 customers found that only 14 percent of issues are fully resolved in self-service, and that even issues customers called very simple resolved only 36 percent of the time (reported by CX Today, 19 August 2024). A later survey of 265 customer service executives found that “nearly 9 in 10 customer service journeys beginning in self-service are ultimately resolved through multiple channels” (CX Dive, 3 September 2025). If almost nine in ten journeys continue elsewhere, then a high containment rate is mostly measuring where people stopped telling you about their problem.

There is a commercial argument against it too. Writing for saas.group, Kalyani Sharma makes the case that deflection quietly costs retention: “Deflection removes friction and removes contact at the same time. For your happiest customers, that’s pure win. For your at-risk customers, you’ve automated away your early-warning system” (29 July 2026). That post is argument rather than data, and says so, but it names the mechanism correctly.

Measure task completion instead: the share of conversations where the thing the user asked for demonstrably happened, verified against your own records rather than the transcript. It is harder to compute and it is the only number that means what you want it to mean.

Instrument before you launch, not after#

Retrofitting this is the difference between a useful week and a week you have to repeat. Six things per turn, recorded from the first day.

The request as the user phrased it, so you can cluster intents rather than guess at them. Every tool called with its arguments, so a wrong call is diagnosable rather than merely visible. The tool’s outcome, distinguishing an error from an empty result, because those are different bugs. Which widgets were rendered, which tells you what the user actually saw. The turn duration split by phase, because most of it is not where you think. And the conversation’s terminal state, which is the only field that requires a product decision and the only one that makes completion computable.

Two things not to log: anything that could carry personal data into an analytics pipeline, and anything on a per-frame basis.

Expect attribution to be broken in a way you cannot fix. SparkToro’s dark-social experiment, using 16 tracked URLs and around 100 recruited panelists across 1,113 visits over ten days, found that “100% of all visits from TikTok, Slack, Discord, Mastodon, and WhatsApp were marked as ‘direct,’ and contained no other referral information” (Rand Fishkin, 27 April 2023). The equivalent inside an app is that you will not know which users came to the assistant because a colleague told them to. Accept a large unattributable bucket rather than reporting it as a tracking failure.

What the logs will not tell you#

Three things, and knowing which three saves you from over-reading the rest.

Why someone stopped. A conversation that ends after two turns could be a satisfied user, a confused one, or a person whose train arrived. The transcript contains no evidence that separates them, and no amount of clustering will produce it. If you need that answer, you need to ask people, and a short prompt shown to a small sample is worth more than a month of inference from logs.

Who did not come. The users who opened the assistant, read the suggested prompts and closed it again leave almost no trace, and they are frequently the largest group in week one. Instrument the open event separately from the first turn, so that ratio is at least visible.

Whether the answer was right. You can verify that an action happened. You cannot verify from the logs that the spoken answer described it accurately, because nothing in the pipeline compares the two. That comparison needs a person reading a sample, or a scored eval, and a week of traffic is exactly when you have the material to build one.

There is also a first-week trap worth naming. The earliest users of any new feature are the curious, the internal staff, and the people who read release notes, which makes them a poor sample of everyone who arrives later. Their intent distribution will be broader and their tolerance for failure much higher than the population you are about to roll out to, so treat week one as a source of shapes rather than of rates.

Turning week one into a change#

The output of the week is a short list, and it should be short.

One or two new tools, drawn from the largest pile in the tail. A handful of description rewrites for tools the model chose wrongly, which is usually the cheapest win available. One schema narrowing for whichever argument the model kept inventing. And a decision about the requests your product does not serve, which is a roadmap conversation rather than an agent one.

Then lock the week’s real transcripts into a repeatable test, so the next schema change is measured rather than argued about. Running the same tasks repeatedly and counting how often they work every time is a more useful number than any average, and week one is when you finally have real tasks to put in it.

Common questions#

What should I measure in the first week of an in-app AI assistant? Intent distribution against what you designed for, failures sorted by cause rather than counted, and task completion verified against your own records. Containment is the number to avoid, because a successful conversation and an abandoned one look the same.

Why is containment or deflection a bad metric? It rises when the assistant works and when the user gives up. Gartner research found only 14 percent of self-service issues are fully resolved, and that nearly nine in ten journeys beginning in self-service continue through other channels.

What do users actually ask an in-app agent? A short head of the requests you expected, then a long tail that is usually larger. The tail splits into requests a tool covers but the model missed, requests your backend could answer but no tool exposes, and things your product does not do.

Where does this sit in the rest of the project? It is the week after launch. The whole stack underneath it, from the speech layer to the tool surface, is the context these numbers are measuring.

How do I sort failed turns? Five buckets: wrong tool chosen, action taken against stale state, unnecessary clarification, tool error, and backend latency. The first three come from the Berkeley Function Calling Leaderboard’s failure modes, and three of the five are fixed in the schema.

Sources#

• Berkeley Function Calling Leaderboard, BFCL V3: Multi-Turn and Multi-Step Function Calling, 19 September 2024, updated 10 December 2024. Accessed 12 September 2026.
• Anthropic, Writing tools for agents, 11 September 2025. Accessed 12 September 2026.
• CX Today, Only 1 in 7 customer service queries resolved with self-service, Gartner study finds, 19 August 2024. Accessed 12 September 2026.
• CX Dive, Self-service, live chat overtake phone and email, 3 September 2025. Accessed 12 September 2026.
• Kalyani Sharma, What AI support deflection quietly costs you in retention, saas.group, 29 July 2026. Accessed 12 September 2026.
• Rand Fishkin, New research: dark social falsely attributes significant percentages of web traffic as direct, SparkToro, 27 April 2023. Accessed 12 September 2026.