Log inBook a demo
GuidesOpens this cluster

Designing an agent conversation people can actually use

The seven decisions that make or break a conversational feature: discoverability, endpointing, barge-in, errors, confirmation, latency and the visual answer.

VVoqal · Engineering11 min read

About this article

Conversation design has a long literature and most of it predates models that can hold a thread. What survived the transition is the part about people: how someone works out what a system can do, what they do when it fails them, and how long they will wait before deciding it is broken.

This is the hub for everything we have written about voice and language. It covers the seven decisions that determine whether a conversational feature gets used twice, and links out to the pieces that go deeper on each.

1. Discoverability, which is the one most teams lose on#

A microphone button is a blank prompt. The user has to invent a request, guess your vocabulary, and interpret any failure as their own mistake. Most agents get exactly one attempt from a given user, and this is where that attempt is spent.

“Ask me anything” is the worst possible affordance because it promises everything and teaches nothing. Nielsen Norman Group’s writing on vague link labels makes the general case against this kind of copy, and it is worth noting honestly that the article argues from UX principle rather than from a controlled study.

Three real tasks in the app's own nouns teach the shape of what is possible. The follow-up chips after a successful turn do the same job with better information.

The fix is three concrete examples in your app’s own vocabulary, refreshed from what this user actually does. And then the cheapest win in the whole design: follow-up suggestions after a successful turn, when the agent knows what the user was trying to do.

An assistant opening screen with a funds figure, a greeting and a list of example requests mixing English and Arabic
The list at the bottom is the onboarding. There is no tutorial and no empty state, because the examples are the state.

2. Knowing when the user has finished#

Endpointing is invisible when it works and is the loudest failure in the product when it does not. Cut someone off mid-sentence and they stop trusting the feature immediately.

The naive approach waits for a fixed quiet period, and it cannot be tuned to satisfy everyone. A user reading a figure off a card pauses mid-utterance; a user saying one word does not.

The lower band is the other direction: what has to happen the moment a user starts talking over the answer.

The better approach scores the words. OpenAI’s realtime documentation describes semantic voice activity detection as a system that “uses a semantic classifier to detect when the user has finished speaking, based on the words they have uttered”, giving a trailing “ummm” a longer timeout than a finished sentence, with eagerness configurable from low to high.

Google’s voice agent guidance adds a case worth handling explicitly: increase the no-speech timeout when the user has something to read before responding. A confirm card is exactly that moment.

3. Barge-in, which is a state machine problem#

Letting users interrupt sounds like an audio feature. It is a concurrency problem, and it is where half-finished implementations show.

Start from the states. Our own phase machine has five: idle, listening, transcribing, thinking and speaking. Transcribing is its own state rather than part of thinking, because the clip has been sent and speech-to-text is in flight, and that is a beat the user can feel. Collapsing it into thinking is how a turn ends up with one undifferentiated spinner covering two very different waits.

The interrupt edge runs from speaking straight back to listening. Everything in flight on the old turn has to be cancelled or suppressed as that edge is taken.

When new speech arrives, three things happen in order and none of them waits for the network: the pending speak task is cancelled, the audio player is stopped, and the phase is set to listening before any transcription comes back. If the phase flips only when the new transcript arrives, the user gets a second of the old answer still talking over them, which reads as the agent ignoring them.

An answer playing aloud, interrupted mid-sentence, with the speak task cancelled and the phase switching to listening
Animation: a spoken answer is cut off mid-word, then speakTask.cancel(), player.stop() and phase = .listening run in that order. Drawn from startVoice() in the SDK, not filmed from the running app.

The subtle failure is the abandoned turn speaking later. A response already in flight when the user interrupted will eventually resolve, and it must not play into the middle of the new question. The way that is enforced is a guard on the playback side: before speaking anything, check the phase, and discard the audio if the machine has moved on. Claim the state first, then suppress the stragglers.

What barge-in costs covers the implementation in detail.

4. The third failure#

Errors are where conversational features lose users permanently, because a user who fails twice concludes that they did it wrong.

Google’s conversation design guidance separates two cases that need different handling. No input means the system “hasn’t heard the user’s response, or the user hasn’t responded by the time the microphone closes”. No match means it “can’t understand or interpret the user’s response in context”. Telling someone “I didn’t catch that” when the real problem is that you did catch it and could not act on it sends them to the wrong repair.

The cap matters more than the wording. A fourth attempt is where the user concludes they are the problem rather than that the feature is.

Two rules from the same guidance are worth pinning above a designer’s desk: “Don’t dwell on or over-explain the error”, and give the user room for self-repair, “allowing the user to repair their turn in the way they think is best”. Google’s Dialogflow documentation recommends a maximum of three no-match or no-input events per page, escalating to a human at the third.

5. Confirmation, tiered so it still means something#

Confirming everything is the same as confirming nothing. If a user taps through four cards a day, the fifth one, which moves money, gets tapped the same way.

Implicit confirmation covers the middle tier: state what you did as you do it, rather than asking permission for something reversible.

Google’s guidance recommends the explicit form as “You mean x?”, repeating only the part that needs checking rather than reciting the whole request. For anything irreversible at value, the card gets a device biometric bound to that single operation, which is the argument made in full for banking.

One rule we enforce in our own stack: the agent never names the authentication method in what it says out loud. The card can show it. Announcing “now confirm with Face ID” tells a room what is about to happen.

6. Latency, and the number we actually measure#

Human conversation is fast. Stivers and colleagues measured turn-taking across ten languages on five continents and found a modal gap of 0 milliseconds between turns, medians ranging from 0 to 300, and a cross-linguistic median around 100. Nothing in this industry is close to that for a turn that calls a real API, and claims to the contrary should be read carefully.

The accented row is ours, measured rather than aspirational. A cold turn is considerably worse, and the cost is dominated by the backend being called.

Jakob Nielsen’s three response-time limits are the practical targets: 0.1 seconds reads as instantaneous, 1 second keeps the flow of thought, 10 seconds is the edge of held attention. Our own warm turn measures about 2.5 to 3 seconds end to end, and a cold one can be much worse, dominated by the tenant’s backend rather than by the agent loop. We used to claim sub-one-second latency on this site. It was not true and it has been retired.

Between one and three seconds, perceived responsiveness is a design problem rather than an engineering one. This is where the five states earn their keep: transcribing and thinking are different waits with different causes, and showing which one the turn is in costs nothing and reads as progress. A visible state change the instant speech ends, then the spoken answer streaming as it arrives, reads as responsive. Three seconds of one undifferentiated spinner does not. The latency playbook covers where the time goes and what can be done about it.

7. What the screen carries#

An answer containing six facts is an answer the user has to memorise. Whitenton’s NN/g analysis of voice interaction makes the structural point: voice-only interfaces force recall where a screen offers recognition.

So the answer splits in two. The spoken sentence stands alone, because someone will hear it with the phone in a pocket. The screen carries what a listener would otherwise have to ask a follow-up question to get, drawn from a fixed catalogue of components the team has already designed and audited.

A dark-theme transactions list with dates, approval status and amounts beneath a spoken summary of the range
A real turn from the demo tenant. The spoken line summarises the range; the rows exist so the user can check it rather than trust it.

The fixed catalogue is doing more work than it appears to. It is what lets you reason about the accessibility of an answer nobody has seen yet, and it is the answer to the reasonable objection that a generated interface cannot be learned or documented.

Language, which is where the craft gets hard#

Everything above assumes the agent understood the sentence. That assumption gets expensive outside English, and it is the part of this discipline with the least written about it.

We run a production assistant that takes turns in English and in Arabic, including dialects, which is the case we can describe from experience rather than from a vendor page. That material has its own hub, covering the long-tail cases, the code-switching problem and the speech-synthesis side.

What belongs here is the part that changes the conversation rather than the model. A user who switches language mid-sentence is making a turn-taking problem, not only a recognition one, because the endpointer and the reply have to agree on which language the turn is in. And an answer synthesised in the wrong register sounds wrong in a way no transcription metric will show you.

The dashboard playground previewing an answer with the entire layout mirrored right to left
The playground with the layout flipped to right-to-left. This is the product's sample data, labelled as such on screen, not a live merchant turn.

Right-to-left layout is the part teams discover late. Mirroring is not a styling pass: numerals, embedded Latin text, icons with direction and swipe affordances all need decisions, and a widget catalogue is where you make them once.

Who this is for, and who it is not#

One piece belongs here as a counterweight. Which situations suit speaking and which suit typing is the narrower question, and a different one from whether you need an agent at all.

On the inclusion side, an agent as an alternative input path covers the curb-cut argument and the WCAG criteria a conversational path touches, and why this matters more for older users covers the cognitive load of navigation along with the adoption data that complicates the easy version of that argument. The WCAG mechanics in depth go further than either.

The short version#

Teach three real tasks instead of promising everything. Score the words rather than the silence when deciding a turn has ended. Let people interrupt, and make sure the abandoned turn stays quiet. Stop at the third failure. Confirm explicitly only where it is hard to undo, and add a biometric only where money moves. Show a state change inside 100 milliseconds even though the answer takes seconds. Make the spoken sentence stand alone, and let the screen carry the rest.

The voice settings reference covers how each of these is configured in the SDK.

Sources#

Filed underVoiceAgentsMultilingualLatency

Next

How a spoken turn is put together, and which parts of it are somebody else's server rather than your code.

Read the voice docs

The rest of Voice

Open the cluster

Latency budgets, barge-in, turn-taking and the parts of a voice pipeline that are somebody else's server rather than your code.

Elsewhere on the map