How fast does a voice agent have to be? A latency playbook

Every voice agent demo is a latency demo. The product being sold is the absence of a pause, and the pause is the only part of the system a user can feel without being told what to look for.

The trouble with the genre is that almost every number in it describes one stage of a pipeline, measured by the company that sells that stage. Adding them up gives you a figure that nobody’s production system has ever produced. This is the general playbook: what the segments are, what each one has actually been measured at by somebody willing to publish a method, and which change moves which segment. Our own breakdown of one real turn, with the profiler output, lives in where the seconds go.

The threshold everyone quotes, and what it really measured#

The figure in circulation is that people answer each other in about 200 milliseconds, so an agent that takes longer feels slow. The underlying work is real and worth reading properly.

Stivers and colleagues recorded informal conversation in ten languages across five continents and measured the offset between one speaker stopping and the next starting. Their 2009 paper in PNAS reports a cross-language median of +100 ms and a cross-language mean of +208 ms, with the language means landing within roughly 250 ms either side of that. Japanese came out fastest at +7 ms and Danish slowest at +469 ms. The paper notes that the 250 ms spread is about as long as it takes to say one English syllable, which is the honest way to read the result: the variation between languages is small enough to be inaudible, and the central tendency is well under half a second.

What the paper does not say is that 200 ms is a threshold below which software feels intelligent and above which it does not. Nobody has published that number for agents, because nobody has run the study. The closest borrowed standard is ITU-T G.114, which Cisco summarises as three bands of one-way delay: 0 to 150 ms acceptable for most applications, 150 to 400 ms acceptable if administrators understand the effect, and above 400 ms unacceptable for general network planning. That recommendation is about transmission delay on a phone call, not about how long an assistant may take to think. Borrowing it is defensible as a sanity check and dishonest as a product claim.

Where the milliseconds go#

A turn is six stages, and only five of them belong to a vendor who benchmarks anything.

Endpointing is the decision that the user has stopped talking. Deepgram’s streaming latency guide puts end-of-turn detection at 100 to 500 ms, transcription at 150 to 300 ms, the full client-side transcript at 200 to 500 ms, and network transit at 20 to 200 ms depending on geography. Those are ranges rather than guarantees, and the same page tells you to read p50, p95 and p99 instead of a single measurement, which is the most useful sentence on it.

Time to first token is where the money is. LiveKit published a table of TTFT measurements on identical requests that spans an order of magnitude: 192 ms for Gemma 4 31B on their own serving, 911 ms for Gemini 2.5 Flash, 966 ms for GPT-5.5, 1,006 ms for GPT-4.1, 1,095 ms for Gemini 3.0 Flash, and 1,876 ms for the same Gemma model served through OpenRouter. The same weights, served two ways, differ by more than a second and a half. Model choice and serving choice are separate decisions and both of them are latency decisions.

Speech synthesis is the stage most often quoted out of context. ElevenLabs states that Flash v2.5 reaches about 75 ms of model inference for typical short inputs, and then does something unusual for a vendor page by listing what the number leaves out: network round trips of 20 to 200 ms, server overhead, audio player buffering that is “commonly 500ms”, and any recognition or model time in a full pipeline. A 75 ms synthesis model sitting behind a 500 ms player buffer produces a 575 ms stage.

The sixth stage is your own backend answering the tool call, and there is no benchmark for it anywhere, because it is your query planner and your indexes.

What we measure, and why it is not sub-second#

Our own numbers are the reason the title of this post changed. A warm Voqal turn lands between 2.5 and 3 seconds end to end. A cold one is between 2.5 and 30 seconds, and the spread is not ours: a cold connection to a tenant’s Model Context Protocol server, including the handshake and the tool listing, has been measured at around 2.6 seconds on a good day and far worse on a bad one. Missing the model’s prompt cache adds roughly four seconds on top. Speech recognition through our provider has varied from 0.5 to 6 seconds between days without any change on our side.

Two things follow. The first is that our engineering effort has gone into removing cold turns instead of shaving warm ones, because the difference between a warm turn and a cold turn dwarfs every micro-optimisation available inside a warm turn. The second is that we no longer publish a sub-second claim. Sub-second is a budget worth designing toward and it is not a description of this stack, and a reader who installs the SDK on the strength of a number will find out within an afternoon.

Six tactics, and the segment each one moves#

Stream between the stages instead of relaying. A pipeline that waits for speech recognition to finalise, then waits for the model to finish, then hands a paragraph to synthesis, pays every stage in sequence. Feeding partial transcripts into the model and the model’s first tokens into synthesis collapses the gaps rather than the stages. This is the largest single change available and it is also the one that makes every other measurement harder, since “how long did recognition take” stops having a clean answer.

Treat endpointing as a decision the system makes, not a timer it runs down. A silence timer is tuned between cutting people off and making them wait, and both settings are wrong for somebody. AssemblyAI’s turn detection write-up describes a model that reads tonality, pacing and rhythm to make the call at roughly 300 ms, which is slower than a bare voice activity detector and considerably better at telling a pause from a finished sentence.

Prime the prompt cache before the user speaks. Anthropic documents the pricing side of caching precisely: cache writes cost 1.25 times base input tokens for the five-minute window, and cache reads cost 0.1 times base input. It does not publish a latency percentage, and neither should anyone quoting it. What we can say is our own: a turn that hits a warm cache runs about 2.5 seconds where the equivalent cold-cache turn ran about 6.7 seconds, on a system prompt carrying tool schemas and tenant context.

Hold tool connections open. The expensive part of a first tool call is rarely the query. It is the connection, the authentication and the tool listing that precede it. A pool keyed by tenant and user, with a long idle window, turns that cost into something paid once rather than once per session. We raised ours from 240 to 600 seconds after measuring how often a returning user fell outside it.

Move the cold work to app launch. Opening the connection and priming the cache when the app starts means the first real turn is a warm turn. This is the single change with the largest effect on how the product feels, because first impressions are formed on first turns and first turns are exactly the ones that used to be cold.

Put the stages in one region. Each hop between a recognition provider, a model provider and your own backend is a network segment in the 20 to 200 ms band. Three providers in three regions is a tax you pay on every turn forever.

What to measure#

Instrument each stage separately and report p95 alongside the median. A median hides the tail, and the tail is what a user remembers, because one four-second turn colours the ten fast ones around it.

Separate cold turns from warm turns in the data. Mixing them produces an average that describes nobody’s experience and moves whenever your traffic pattern changes.

Measure the moment the user first sees something, not only the moment audio starts. A turn that shows a listening state, then a transcript, then a thinking state, then speech is doing more for perceived speed than any of the tactics above. Our SDK runs five states for exactly this reason: idle, listening, transcribing, thinking and speaking, with transcribing kept separate so the interface can show that the words were heard before the answer exists. What the first 300 ms should show covers the design side of that, and the cost of letting a user interrupt covers the case where the user interrupts and the whole budget resets. The wider question of how a spoken conversation should be shaped sits in designing an agent conversation people can use.

If you want the segment-by-segment numbers for a real deployment instead of a vendor range, how the voice path is configured describes what the SDK reports per turn and how to read it.

Sources#

• Stivers, T. et al., “Universals and cultural variation in turn-taking in conversation”, PNAS, 2009. Accessed 12 September 2026.
• Cisco, “Understand Delay in Packet Voice Networks”, summarising ITU-T G.114. Accessed 12 September 2026.
• Deepgram, “Measuring Streaming Latency”. Accessed 12 September 2026.
• LiveKit, “Latency-optimized inference: Gemma 4 on LiveKit”. Accessed 12 September 2026.
• ElevenLabs, “Understanding latency”. Accessed 12 September 2026.
• AssemblyAI, “Voice agent turn detection”. Accessed 12 September 2026.
• Anthropic, “Prompt caching”. Accessed 12 September 2026.