The first 300 milliseconds of an agent turn

There is a moment in every agent turn where the user has asked and nothing has happened yet. It is short, it is entirely yours, and most implementations spend it showing a spinner.

That moment is worth more attention than the model choice, because it is the only part of the turn your client can change.

What the user should see before the model has said anything#

Two things, and neither depends on the answer. The visible state should change in the same frame as the tap, so the user knows the input landed. Then the interface should name the phase the turn is genuinely in, which for the first fraction of a second is listening, and a moment later is transcribing, thinking, or whichever tool is running.

Everything in that window is decided by code you wrote. Nothing in it is waiting on inference.

This matters more for an agent than for an ordinary network call, because the shape of the answer is unknown until late. When an agent draws its answer as one of your own native views, the widget kind is chosen by the model after the tool returns, which means the client genuinely cannot know at 200 milliseconds whether a card or a list is coming. Loading states built on the assumption that you know are the most common defect in this whole category.

The phases of a turn, and which ones you control#

Our SDK models a turn as five states: idle, listening, transcribing, thinking, speaking. Transcribing is worth having as its own state rather than folding it into thinking, because it is the one segment whose duration you cannot predict and did not cause, and a user told the agent is still transcribing is being told something true.

A turn is a sequence of handoffs, and you own the ends. Touch feedback and the phase machine are yours. End-of-speech detection is a policy you set, against a provider’s signal. Transcription and the model’s first token belong to services you do not run. The tool call goes to your own backend, so its slow tail is your slow tail. Validating the spec and drawing the view is yours again, and takes almost no time.

The inference layer is where most published latency work happens, and it is worth reading rather than competing with. LiveKit’s latency-optimised inference post, published in February 2026, gives time-to-first-token figures across six configurations, from 192 ms on their own stack to 911 ms for Gemini 2.5 Flash, 966 ms for GPT-5.5 and 1,876 ms routing the same Gemma 4 model through OpenRouter. Their time-to-first-speech numbers run from 354 ms to 4,120 ms across the same set. Those are vendor-published self-benchmarks, so read them as a spread rather than a ranking, and note the spread is what matters here: the segment you do not own can vary by an order of magnitude for reasons that have nothing to do with your code.

That is the argument for spending effort on the segment you do own. Where the seconds actually go in a production turn is a separate measurement, and the answer there is also mostly not your code.

Streaming the answer without streaming the widgets#

Text and structure want different treatment. Speech and prose should stream, because a partial sentence is useful and a user can start listening before the sentence is finished. A widget should not stream, because a half-populated balance card shows a number that is wrong until it is right.

The rule that works is to stream the spoken part and hold the render until the spec validates. That gives the user something within a few hundred milliseconds of the model starting, and gives them a correct view rather than a settling one. The contract that makes this possible puts the speech and the spec in one response with the speech first, deliberately.

Skeletons that tell the truth#

A loading state is a promise about what is coming. Most skeleton implementations make a promise the turn has not yet earned.

A shimmering card outline says a card of about this size is arriving. In an agent turn you do not know that yet, because the widget kind is chosen by the model after the tool returns. When a list arrives where the skeleton promised a card, the layout jumps, and a layout jump under a thumb is worse than having shown nothing.

Naming the phase avoids the problem entirely, because a phase is a fact about the past. Vercel’s AI SDK encodes the same idea in its tool-part states, input-available, output-available and output-error, which describe where the call has got to rather than what the result will look like.

When to show the tool call, and when to hide it#

Show it when the tool name tells the user you understood the question. “Looking up your transactions” is reassurance, because the user’s real anxiety during the wait is not duration, it is whether the agent understood. Hide it when the name would leak an internal concept, and hide it when the call returns fast enough that the label would flash and vanish, which is worse than no label.

The rule of thumb that survives contact with real users: show the tool phase if the call typically takes longer than about half a second, and use words from the user’s vocabulary rather than the function name.

What to do when it is genuinely slow#

Sometimes a turn takes several seconds and no amount of phase naming fixes it. Three things help, in order of how much they help.

Keep the user informed with a phase change rather than a progress bar, because a progress bar implies you know the denominator and you do not. Let them interrupt: if speech is playing and the user starts talking, the agent should stop immediately, which turns a long turn into a conversation they are still driving. And if a tool call is the slow part, say which one, because a user who knows the app is waiting on a bank blames the bank.

This is also the case where a warm connection matters most. The submitter of a March 2026 Hacker News thread on treating Markdown as a protocol for generative UI described the failure mode they were designing against in their own words: “if the agent creates a complex mini app, the user might have to wait 30 seconds. That’s 30 seconds without feedback.” Their proposed fix was incremental execution, so that “API calls start, UIs render, and errors surface while the LLM is still streaming tokens.” That is the author’s framing of their own project rather than a measured result, and the thirty seconds is a hypothetical. The instinct behind it is sound and it is the same instinct as streaming the speech first.

The measurements that matter#

Three numbers, and only one of them is time to first token.

Time to first visible change, measured from the touch event. This should be a single frame and it is the number most teams never look at, because it is not in any vendor’s dashboard.

Time to first audio, for voice turns, measured from end of speech. This is the one the user experiences as the agent’s reaction time, and it includes endpointing, which is a policy you set rather than a latency you suffer.

Time to first widget, measured from the same point. This will always be later than first audio if you stream speech first, and the gap is the thing to keep stable, because a user who hears an answer and then waits four seconds for the card stops trusting the card.

What “fast enough” means, and whose fast we are borrowing#

The honest anchor for conversational expectation is human, not machine. Stivers and colleagues, in Universals and cultural variation in turn-taking in conversation (PNAS, 2009), recorded conversation in ten languages across five continents and found that “the response timings for each language, although slightly skewed to the right, have a unimodal distribution with a mode offset for each language between 0 and +200 ms, and an overall mode of 0 ms.” Medians ranged “from 0 ms (English, Japanese, Tzeltal, and Yélî-Dnye) to +300 ms (Danish, ‡Ākhoe Hai‖om, Lao)”, with a cross-linguistic median of +100 ms.

That is a fact about humans answering humans, and no agent stack answers in 0 ms. Quoting it as a target would be silly. Quoting it as an explanation is fair: it is why a one-second pause before any response feels long to a person who has never thought about latency in their life. The comparison they are making is to every conversation they have ever had. Other agents do not come into it.

Which is also why the first 300 milliseconds carry so much weight. You cannot answer in 0 ms. You can react in one frame, and a reaction is what the other party in a conversation actually provides at the zero mark. Conversation design for the rest of the turn and the latency playbook itself pick up where this leaves off.

Common questions#

What should an app show while an AI agent is thinking?#

Change the visible state in the same frame as the input, then name the phase the turn is in, and name the tool once one is running. Our own phase machine has five states, idle, listening, transcribing, thinking and speaking, and each is a fact about where the turn has already got to. Avoid skeletons shaped like a specific widget, because the widget kind is not known until the model has chosen it.

How fast does an AI assistant need to reply?#

There is no published threshold for agents. The nearest honest anchor is human conversation, where Stivers and colleagues measured an overall response-offset mode of 0 milliseconds across ten languages, with per-language modes between 0 and 200 milliseconds. Users are comparing your agent to that, not to another product.

Should you stream the agent’s answer?#

Stream the spoken and written answer, because a partial sentence is useful. Do not stream a widget, because a partly populated view displays a number that is wrong until the last field lands. Sending the speech first and the structure second gives you both.

What is time to first token, and does it matter on mobile?#

Time to first token is how long the model takes to emit anything after the prompt is sent. It matters, but it is not yours: LiveKit’s published figures for the same class of task span 192 to 1,876 milliseconds depending on the stack. Time to first visible change, which is yours, is usually the more actionable number.

Sources#

• Tanya Stivers, N. J. Enfield, Penelope Brown, Christina Englert, Makoto Hayashi, Trine Heinemann, Gertie Hoymann, Federico Rossano, Jan Peter de Ruiter, Kyung-Eun Yoon and Stephen C. Levinson, Universals and cultural variation in turn-taking in conversation, PNAS, 2009, volume 106, issue 26, pages 10587 to 10592.
• LiveKit, Latency-optimized inference: Gemma 4 on LiveKit, February 2026. Vendor-published self-benchmark.
• Vercel, Generative User Interfaces, AI SDK documentation, for the input-available, output-available and output-error tool states.
• Hacker News, I turned Markdown into a protocol for generative UI, submitted 19 March 2026. Quotes are from the submission’s own author.