Where the seconds go in an in-app agent turn

A turn in our production assistant once took thirty seconds. We spent a while assuming it was us.

It was not. The numbers below are from our own instrumentation of a live merchant assistant, including the ones that make our work look unnecessary, because those are the ones that would have saved us time if somebody else had published them first.

The measured breakdown#

Two caveats before anyone quotes these. They are from one production deployment, a payments assistant against a partner’s tool backend, so they are a shape rather than a benchmark. And we do not publish a fine-grained decomposition of the warm 2.5 seconds, because we have not measured one we trust, and inventing a breakdown is how latency posts become fiction.

The layers, and who owns each one#

Four parties are involved in a turn and only one of them is your application.

Your own code owns the client, the phase machine, the tool call into your API and the render. On a warm turn this is the part that costs least, which is uncomfortable if you have spent a sprint on it.

The inference provider owns the model’s response time, and that number moves without telling you. LiveKit’s published comparison from February 2026 gives time-to-first-token across six configurations ranging from 192 ms on their own stack to 1,876 ms routing the same model through OpenRouter. Those are their own self-benchmarks, and they are measuring a layer below this post. We cite them because the spread is the point: the same model can differ by an order of magnitude on first token depending on how it is reached.

The speech provider owns transcription, and this was our most surprising finding. Our speech recognition has varied between roughly half a second and six seconds for comparable audio, day to day, with no change on our side. We chased it as a regression twice before accepting that it was weather.

That variance is why our SDK carries transcribing as its own state, rather than folding it into thinking. The turn has five: idle, listening, transcribing, thinking, speaking. A segment you cannot predict and did not cause is a segment worth naming on screen, because the alternative is a user watching a thinking indicator for six seconds while nothing is being thought.

And the tool backend owns the largest single cost on a cold turn, which is the next section.

What a cold connection actually costs#

The first turn after a restart has to open a connection to the tool backend, complete a handshake, and ask what tools exist. Measured directly against the partner’s endpoint, an initialize, a list_tools and one tool call came to about 2.6 seconds. Going through the client library’s own tool discovery came to about 2.4 seconds. On a bad day the connection phase alone reached thirty seconds.

Nothing in our code changes that. The one thing we can do is not pay it while a user is waiting, which is what the connection pool is for.

The thirty-second turn is worth describing, because the diagnosis took longer than the fix. Our first assumption was a regression in our own request path, so we bisected. Our second was the model, so we swapped it. Neither moved the number. What eventually settled it was running the handshake from a plain script with no part of our stack in the path, and watching it take the same thirty seconds. The partner’s endpoint was having a bad hour. We had spent two days looking in the one place the problem could not be, which is the failure mode this entire post exists to prevent.

Raising the idle timeout was the cheapest win in the whole exercise. It is also the least interesting, which is roughly the rule for latency work.

The prompt cache is the single biggest lever#

The clearest measured pair we have is the same turn with and without a warm prompt cache: 6.7 seconds against 2.5. Roughly four seconds of a slow turn was the model re-reading a system prompt it had already been shown.

The reason the prompt is large enough for this to matter is worth knowing, because it is a cost you control. Anthropic document that enabling tools adds a system prompt of its own, costing between 264 and 804 tokens depending on the model and on whether tool choice is forced. Your own tool schemas sit on top of that, on every turn, whether or not the model calls any of them. How you shape those schemas is therefore a latency decision as well as a correctness one.

What prewarming buys, and what it cannot skip#

Prewarming opens the connection and primes the prompt cache at app launch, so the first real turn finds both ready. It removes the cold connect and the cache miss from the user’s experience of the first question.

It does not make a turn faster. The work still happens; it happens while nobody is waiting. And it cannot skip the live data fetch, because the assistant’s opening state reads a current snapshot, and a stale snapshot shown instantly is a different feature with a different set of problems.

The thing we thought would help and did not#

We were about to build a tool-schema cache, so that a turn would not have to ask the backend what tools exist. It sounds obviously right.

Measuring first killed it. Listing the tools was about 1.3 seconds of the 2.6 second sequence, and the turns where it mattered were exactly the turns already paying for a cold connection. Fixing the connection lifetime removed that cost and more, for a configuration change.

We are writing this down because the week we did not spend is the useful part, and nobody publishes the optimisation they abandoned.

What “fast enough” means#

There is no published threshold for agent turns, so people borrow one, and the two borrowings in circulation both need a warning label.

The first is the telephony figure. Cisco’s summary of ITU-T G.114 gives three bands: 0 to 150 ms “acceptable for most user applications”, 150 to 400 ms “acceptable provided that administrators are aware of the transmission time”, and above 400 ms “unacceptable for general network planning purposes”. That is one-way transmission delay on a voice network, measured for connections with echo controlled. It is a budget for moving audio, not a budget for thinking. Using it as an agent’s response target is a category error, and it is repeated constantly.

The second is human conversation, and it is the honest one as long as you say what it is. Stivers and colleagues, in Universals and cultural variation in turn-taking in conversation (PNAS, 2009), recorded ten languages across five continents and found “a mode offset for each language between 0 and +200 ms, and an overall mode of 0 ms”, with medians running “from 0 ms (English, Japanese, Tzeltal, and Yélî-Dnye) to +300 ms (Danish, ‡Ākhoe Hai‖om, Lao)”.

Our warm turn is roughly ten times the upper end of that range. That gap is the honest headline, and it is why what the interface does in the first fraction of a second is the only part of the gap you can close this quarter.

Systems that have closed part of it did so by changing the conversation rather than the inference. A deployed spoken dialogue system described in Duplex Conversation (arXiv, May 2022) reports that “online A/B experiments show that the proposed system can significantly reduce response latency by 50%”, achieved through user state detection, backchannel selection and barge-in detection. That was a telephone-based customer service deployment at Alibaba rather than an in-app assistant, so the transfer is partial. The mechanism transfers completely: none of those three are model optimisations.

What we would tell you to measure first#

Measure the cold and warm cases separately, because averaging them produces a number that describes no real turn. Measure your tool backend directly, outside your own stack, before you optimise anything inside it. Measure the cache hit rate on your system prompt, because that was our largest single lever and it is invisible unless you look for it. And measure your speech provider across days rather than within one session, because a provider that is fast this afternoon may not be fast on Monday.

The general version of this advice, with the playbook for each layer, goes further than we can here. The specific version is short: most of your latency is somebody else’s, and the fix is usually to stop paying for it while a user watches. That starts with pointing the agent at your backend carefully and keeping that connection alive.

Common questions#

Why is my AI assistant slow?#

Usually because a turn is paying a cost that should have been paid earlier. The two large ones are opening a connection to the tool backend, which we have measured between 2.5 and 30 seconds, and a prompt cache miss, which cost us about 4 seconds. Neither is in application code.

What is a cold start for an AI agent?#

It is the first turn after the runtime has no open connection to the tools and no warm prompt cache. That turn pays for the handshake, tool discovery and a full re-read of the system prompt. Subsequent turns reuse all three, which is why the first question of a session feels different from the second.

How much does prompt caching help?#

In our measurement of one production turn, 6.7 seconds became 2.5 seconds. The size of the win depends on how large your system prompt and tool schemas are, and Anthropic document that enabling tools alone adds between 264 and 804 tokens before any of your own schemas are counted.

How fast should a voice agent respond?#

No credible published threshold exists for agents. Human conversation has an overall response-offset mode of 0 milliseconds across ten languages, which is what users compare you to and is not reachable. The telephony figure of 150 milliseconds that circulates is a one-way audio transmission budget and does not apply.

Sources#

• Tanya Stivers and colleagues, Universals and cultural variation in turn-taking in conversation, PNAS, 2009, volume 106, issue 26.
• Cisco, Understanding Delay in Packet Voice Networks, for the ITU-T G.114 one-way transmission delay bands.
• LiveKit, Latency-optimized inference: Gemma 4 on LiveKit, February 2026. Vendor-published self-benchmark.
• Anthropic, Tool use overview, for the per-model tool-use system prompt token table.
• Ting-En Lin, Yuchuan Wu, Fei Huang, Luo Si, Jian Sun and Yongbin Li, Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems, arXiv:2205.15060, May 2022.
• Our own instrumentation of a production merchant assistant, 2026. One deployment, one tool backend; reported as a shape rather than a benchmark.