What barge-in really costs, and how we pay for it

Interrupting an assistant mid-sentence sounds like a feature you get for free once you have a microphone. It is a coordination problem across two systems that do not know about each other, and when it goes wrong the user experiences it as the assistant ignoring them, which is the single most irritating thing a voice product can do.

Two different problems, both called barge-in#

The first is deciding that the user has started talking and means it. That happens on the server, from the audio stream, and it is a signal-processing and classification problem.

The second is making the agent actually stop. That happens on the device, and it is a concurrency problem. An audio buffer is playing, a speech synthesis request may still be in flight, and a state machine somewhere believes the agent is speaking. All three have to change in the right order, in under the time it takes a person to notice.

Papers and vendor documentation cover the first half thoroughly. The second half is almost entirely undocumented, and it is where our own bugs lived. It sits underneath everything in the conversation design playbook, which is the place to start if you are designing the turn rather than implementing it.

The server half, and why it is usually the rude part#

Turn detection has real options now, and they behave differently.

Voice activity detection is the classical approach and is configured with three numbers. OpenAI’s realtime documentation exposes a threshold, described as an “activation threshold (0 to 1). A higher threshold will require louder audio to activate”; a prefix padding, the “amount of audio (in milliseconds) to include before the VAD detected speech”; and a silence duration, the “duration of silence (in milliseconds) to detect speech stop” (OpenAI). Those three numbers are where most perceived rudeness comes from. Short silence duration and the agent talks over a thinking pause. Long silence duration and it feels sluggish.

Semantic turn detection is the newer answer, and it is better on exactly the case VAD handles worst. OpenAI’s semantic mode “uses a semantic classifier to detect when the user has finished speaking, based on the words they have uttered”, scoring audio on the probability that the user is done and extending the timeout when the words suggest hesitation. It is tuned with an eagerness setting where low “will let the user take their time to speak” and high “will chunk the audio as soon as possible”.

LiveKit’s turn detector takes the same position from the acoustic side, running “semantic and acoustic prediction on top of VAD” and analysing “the words with acoustic cues like intonation and rhythm”, which lets it decide without waiting for a transcript (LiveKit, 30 June 2026). Their documentation is the best public configuration reference for this layer and we are not going to write a better one.

The idea is older than any of it. Ström and Seneff’s 2000 MIT paper on intelligent barge-in describes a three-phase design that still maps onto every modern system: detection, where “a 50 ms frame rate is used, and a frame is marked as speech if both energy and periodicity exceed their respective thresholds”; verification, using a threshold on the recogniser’s confidence score plus a requirement that the language understanding component can actually produce an interpretation; and recovery, for when verification decides the interruption was not real. It is a design paper and reports no latency or accuracy figures, which is worth saying when citing it.

The most useful deployment result comes from Alibaba’s Duplex Conversation, presented at KDD 2022, which split the problem into “user state detection, backchannel selection, and barge-in detection” and reported that the deployed system “can significantly reduce response latency by 50%” (Lin et al.).

The client half: three things, in one order#

When the server says the user has the turn, three operations have to happen on the device, and the order is the whole design.

Cancel the speech task#

If a synthesis request is in flight, cancel it. Not “ignore its result when it arrives”, because the result arriving is what restarts playback two seconds later when the user is mid-sentence. The task handle has to be held somewhere cancellable, and cancellation has to be the first thing the interruption path does.

Stop the player#

The audio player is holding a buffer and will finish it unless told otherwise. Stopping is synchronous and fast, and it is the part users actually perceive, so it goes early.

Claim the phase before anything else can#

This is the one that is easy to miss. The moment the microphone opens, the phase becomes listening, and it becomes listening before either of the two operations above can fail, return late, or be overtaken. Every other path that wants to speak has to check the phase first and stand down if it is listening.

In our own experience model this is a small amount of code and it took several attempts to get right. Starting voice input cancels the speak task, stops the player, and claims the listening phase; and the speech paths are guarded so that neither a queued response nor a late synthesis result can play while the phase is listening.

The race conditions, named#

Three of them, all of which we have shipped at some point.

The late synthesis result. Audio arrives after the user has started speaking, because the request was not cancelled, only its result discarded somewhere further down. The agent talks over the user with a sentence about the previous question.

The phase overwrite. Two paths write the phase, one of them asynchronously, and the slower one wins. The microphone is open but the machine believes the agent is speaking, so the guard that should suppress playback does not fire.

The stale turn. The user interrupted, asked something new, and the answer to the previous question arrives first because it was already in flight. The fix is not on the audio path at all: every turn carries an identity, and a response whose identity is not the current turn is discarded before it reaches the player.

Why the phase machine has to be the only source of truth#

The temptation is to let each component track its own state. The player knows whether it is playing. The synthesiser knows whether a request is open. The microphone knows whether it is capturing. Three booleans, each correct about its own component.

They will disagree. A phone is a device where audio focus can be taken by a phone call, a route can change when a headset disconnects, and any request can be slow. Three sources of truth mean three opportunities to be in a state no one designed.

One phase, written in one place, with every audio-producing path reading it before it acts. In ours there are five: idle, listening, transcribing, thinking and speaking.

The fifth one is worth arguing for, because it is the one most designs collapse. Transcribing is its own state, held while speech-to-text is in flight, and it is separate from thinking. They feel identical to a developer reading the code and they are not: transcribing owns audio the user has already produced, so an interruption arriving during it has to discard work that is already paid for, while an interruption during thinking only cancels a model call. Merging them into one “busy” state is how you end up with a transcript from a previous utterance surfacing after the user has moved on.

There is one other detail worth copying: for a voice turn, the speaking phase is entered when audio actually starts playing rather than when synthesis is requested. Flipping it early opens a window where the machine says speaking, nothing is audible, and an interruption in that window does nothing visible, which users read as the assistant ignoring them.

How do you know it worked?#

Barge-in is hard to test because the failure is a timing window, and a passing manual test proves one interleaving.

Three checks that catch real regressions.

Assert on audio, not on state. A test that confirms the phase became listening passes while sound is still coming out of the speaker. Assert that the player reports stopped, and that no further buffer is scheduled.

Interrupt at the worst moment. Not mid-sentence, which is easy, but in the window between the synthesis request and the first audio frame. That is the window where a late result overwrites a correct state.

Interrupt twice quickly. Two interruptions inside a second is how the phase-overwrite bug surfaces, because the second one arrives while the first is still settling.

None of this needs a device farm. It needs the audio layer behind a protocol you can drive from a test, which is worth arranging before you need it rather than after. What the user sees during the window you are testing is a separate design problem, and the two interact: a skeleton that appears the instant the mic opens makes a slow cancellation much less noticeable.

What we would concede to the server if we could#

The honest version of this post ends with a concession. If we could move one thing to the server, it would be the decision itself, because a semantic turn detector with access to the words is better at distinguishing a thinking pause from a finished sentence than anything a client can compute from energy alone. Google’s Live API documents the same capability from the other side, noting that “users can interrupt the model at any time for responsive interactions” (Gemini Live API).

What cannot move is the execution. The cancellation, the player, and the phase live where the audio lives. A perfect server decision delivered to a client that keeps talking for another 400 milliseconds is a rude assistant with excellent classification.

There is also a constraint worth naming because it shapes the whole design: our speech recognition provider transcribes per clip rather than emitting true live partials, so the client accumulates a clip and transcribes at end of speech. That makes the interruption decision a client-side one more often than we would like, and it is the main reason the client half of this post is as long as it is. Provider choice here is constrained by more than latency, because the languages and dialects you have to serve narrow the field considerably.

For what a reasonable target looks like, human conversation is the honest anchor. Stivers and colleagues, studying ten languages across five continents, found response-offset modes between 0 and +200 ms with “an overall mode of 0 ms”, and medians ranging “from 0 ms (English, Japanese, Tzeltal, and Yélî-Dnye) to +300 ms (Danish, ‡Ākhoe Hai‖om, Lao)”, concluding that all the languages tested show “a general avoidance of overlapping talk and a minimization of silence between conversational turns” (PNAS, 2009). That is a fact about people, not about machines, and quoting it as a machine budget is a category error. It is still the right thing to measure yourself against. A warm agent turn in our runtime settles around 2.5 to 3 seconds, which is nowhere near it, and where those seconds go is a separate and more uncomfortable post.

Common questions#

What is barge-in in a voice assistant? The ability to interrupt the assistant mid-sentence and have it stop and listen. It is two problems: a server-side decision that the user has taken the turn, and a client-side execution that cancels in-flight speech, stops the player and claims the listening state.

Why does my voice assistant talk over me? Usually a late synthesis result. The request was not cancelled, only its result discarded somewhere downstream, so audio arrives after the user has started speaking. The second most common cause is two code paths writing the same state, where the slower one wins.

What is the difference between VAD and semantic turn detection? Voice activity detection decides from the audio signal using a threshold and a silence duration. Semantic turn detection classifies whether the user has finished based on the words spoken, extending the timeout after hesitant phrasing. The second handles thinking pauses much better.

How fast does an agent have to stop talking? Fast enough that the user does not hear a word after they started speaking. For the surrounding turn, human conversation is the reference: response offsets cluster around a mode of 0 ms across ten languages, which no current agent stack approaches.

Sources#

• OpenAI, Realtime voice activity detection. Accessed 12 September 2026.
• LiveKit, Turn detection and interruption handling, 30 June 2026. Accessed 12 September 2026.
• N. Ström and S. Seneff, Intelligent barge-in in conversational systems, MIT Laboratory for Computer Science, 2000. Accessed 12 September 2026.
• Lin et al., Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems, KDD 2022. Accessed 12 September 2026.
• Google, Gemini Live API. Accessed 12 September 2026.
• Stivers et al., Universals and cultural variation in turn-taking in conversation, PNAS, 2009. Accessed 12 September 2026.