Running ten dialects of one language through one pipeline

Every speech stack ships with an assumption baked so deep that nobody states it: one user speaks one language, and that language has one spoken form. Both halves are false for a large share of the world’s users, and the failures do not look like language failures. They look like the assistant being bad.

Arabic is the worked example throughout this post because it is the case we run in production and the one where the gap is widest. The engineering is not about Arabic. Swiss German against standard German, Cantonese against written Chinese, Haitian Creole against French, and most of South Asia produce the same three problems with different vocabulary.

The assumption that breaks#

Ask a speech vendor which languages they support and you get a list. Gemini’s Live API documents that you can “converse in 70 supported languages” with real-time translation (Google). That is a real capability and it is also a list of labels, and labels hide two things.

A label is a written standard. The thing arriving at your microphone is a spoken variety, which may differ from the standard in its sound inventory, its vocabulary and its grammar.

And a label is singular. Users are not. A person will start a sentence in one language, name a product in another, and finish in the first, inside four seconds.

When the written and spoken forms diverge#

Most languages have some gap between how they are written and how they are spoken. In some it is small enough to ignore. In others it is large enough that a model trained on the written form transcribes the spoken one badly.

Munsit, a vendor specialising in this for Arabic, states the mechanism plainly: Arabic speech recognition is hard because people “write in Modern Standard Arabic (MSA)” but “speak in regional dialects”, and “differences in pronunciation (like qaf), vocabulary, and grammar mean a single generic Arabic model can’t reliably serve all three dialect groups” (Munsit, updated 17 July 2026). NeuralSpace, which ships dialect-specific models for Modern Standard, Gulf, Egyptian and Levantine Arabic, makes the same argument from the product side, writing that the differences between the standard and the regional forms “underscore the need for speech-to-text technologies to evolve and accurately capture linguistic nuances” (NeuralSpace). Neither publishes a word error rate per dialect, which is a fair description of the state of the public evidence.

The practical consequence is that training data volume does not rescue you. Whisper was trained on “680,000 hours of multilingual and multitask supervision”, and its authors describe the result carefully: “when compared to humans, the models approach their accuracy and robustness” (Radford et al., 6 December 2022). Approach. Not match, and not exceed, and that careful verb applies to the aggregate. A variety that is two percent of the training set does not get the aggregate’s performance.

Code-switching defeats detect-then-transcribe#

The standard architecture for multilingual speech is two stages: identify the language, then transcribe with the model for that language. It is clean, it is what most SDKs expose, and it cannot represent the input.

Consider an utterance that begins in one language, contains an English product name and a number, and returns. Language identification has to produce a single answer for a stretch of audio in which the answer changes twice. Whichever it picks, part of the utterance is transcribed by the wrong model, and the part it gets wrong is usually the part carrying the entity you needed.

Three things help, in order of how much they help.

Use a recogniser that does not require a language decision up front. Multilingual models that transcribe without a committed language label degrade gracefully on switched audio instead of failing at the switch point.

Give the recogniser your vocabulary. Product names, merchant names and the handful of domain terms your users say constantly are exactly the tokens that get mangled, and they are also a short, enumerable list you already have in a database.

Do not correct the transcript before the model sees it. A normalisation step that rewrites a spoken form into a standard one throws away the evidence the agent could have used, and it is where a correct transcript becomes a wrong one.

The recognition side, in production#

Three changes mattered more than anything else we tried.

We transcribe per clip rather than streaming partial hypotheses, because our provider transcribes a completed utterance rather than emitting true live partials. That is a constraint rather than a preference, and it pushes the end-of-speech decision onto the client, which is its own engineering problem.

We stopped treating a low-confidence transcript as a failure. A confident wrong transcript is far more dangerous than an uncertain one, particularly when the next step is a tool call that moves money. The uncertain case should reach a confirm card showing the parsed arguments, where the user can see that the amount is wrong before anything happens.

And we stopped assuming the reply language from the request language alone. Users switch between turns as well as within them, and a user who asked in one language and then typed in another expects the second answer in the second language.

The generation side#

Recognition gets the attention. Generation is where you decide whether everyone can understand the answer.

The rule we settled on is to reply in the standard written form of the language rather than in any one regional variety. In Arabic that means Modern Standard Arabic, always, even when the user spoke a dialect. It sounds slightly formal to every listener and it is intelligible to all of them, which is the correct trade for an assistant that a whole market uses. Picking one regional variety to generate in means picking a subset of your users to sound native to and everybody else to sound foreign to.

Two mechanical points support that.

Formatting belongs to the client, not the model. The model emits a value and a currency code; the render spec carries it; and the client decides numerals, separators and direction from the user’s locale. This is what stops a right-to-left layout rendering a mirrored amount.

And synthesis quality varies far more by language than recognition does. Budget for the possibility that your speech synthesis is the weakest component in the chain for your second language, and listen to it in the target variety before you decide the pipeline is finished.

How to evaluate your own coverage#

This is the part where the industry is weakest, and we include ourselves.

An aggregate word error rate across a language is close to meaningless when the varieties differ as much as they do. A vendor claiming support for a language is claiming something about a weighted average whose weights they have not published. Ask three questions: which varieties are in the evaluation set, what the error rate is for each one separately, and how many hours of each are in the training data.

Public benchmarks exist and are worth using. VoxArabica covers dialect identification across 17 dialects plus Modern Standard Arabic, with recognition models fine-tuned on standard, Egyptian, Moroccan and mixed data, and zero-shot coverage for the rest (Waheed et al., ArabicNLP at EMNLP 2023). Its value is the structure as much as the scores: it treats identification and recognition as separate measurable things, which is exactly the split a buyer needs.

We do not publish a per-variety word error rate for our own pipeline, because we have not measured one to a standard worth publishing. Building that evaluation set is real work, and until it exists the honest claim is that we run one pipeline in production across users who speak many varieties, not that we have quantified how well it serves each one. Anyone claiming otherwise without a published set and per-variety numbers is describing a weighted average too.

What transfers to any multilingual product#

Strip out the specifics and five rules remain, and they hold wherever the written and spoken forms diverge.

Never commit to a language for a whole utterance. Choose per turn at the latest, and prefer a recogniser that does not require the choice at all.

Evaluate per variety and publish the split, internally at minimum. An aggregate hides exactly the failure you will get complaints about. The procedure for measuring that split is written up separately, and it is the same procedure in any language whose speakers do not all speak it the same way.

Generate in the form everyone understands, even when it sounds slightly formal to everyone.

Keep formatting in the client, driven by the user’s locale rather than by anything the model said.

And treat a low-confidence transcript as a reason to show the user what you heard, rather than as a reason to guess.

The last piece of context worth holding: input on a phone is slow everywhere, and slower in scripts whose input methods are harder. A study of 37,370 participants measured an average mobile typing speed of “36.2 WPM with 2.3% uncorrected errors”, and noted that with one or two fingers people type “about 70% as fast on mobile devices as on full desktop keyboards” (Palin et al., MobileHCI 2019). That is a measurement of English typing, so read it as the optimistic case rather than the general one.

Common questions#

How do you handle multiple dialects of one language in a voice agent? Do not identify the language and then commit to it. Use a recogniser that tolerates switching, feed it your own product and merchant vocabulary, skip any normalisation step before the model sees the transcript, and generate in the standard written form so every speaker understands the reply.

Why does speech recognition fail on dialects? Because models are trained mostly on the written standard and on the varieties with the most available audio. A variety with different pronunciation, vocabulary and grammar is under-represented in training, and aggregate accuracy figures hide that.

Can a speech model handle code-switching within one sentence? A single-language pipeline cannot, because it has to pick one answer for a stretch of audio where the answer changes. Multilingual models that transcribe without a committed language label degrade more gracefully at the switch points, which is where the entities usually are.

How should I evaluate a vendor’s dialect coverage? Ask which varieties are in their evaluation set, ask for the error rate per variety rather than in aggregate, and ask how many hours of each variety are in training. A single number for a language is a weighted average with undisclosed weights.

Sources#

• Google, Gemini Live API. Accessed 12 September 2026.
• Munsit, Arabic dialect speech recognition: Gulf, Egyptian, Levantine, updated 17 July 2026. Accessed 12 September 2026.
• NeuralSpace, Introducing dialectal speech-to-text models for Arabic. Accessed 12 September 2026.
• Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision, 6 December 2022. Accessed 12 September 2026.
• Waheed et al., VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System, ArabicNLP at EMNLP 2023. Accessed 12 September 2026.
• Palin et al., How do People Type on Mobile Devices?, MobileHCI 2019. Accessed 12 September 2026.