How to evaluate dialect coverage in a speech stack

Every speech vendor publishes a language list, and every language list is a claim about a training corpus. It says a model has seen the language. It says nothing about whether the model works on the variety your users actually speak, and for some languages the distance between those two things is enormous.

This is the method for finding out. Arabic runs through it as the worked example because it is the hardest case in wide commercial use, and because we run an agent in production in it. The procedure transfers to Hindi, Spanish, Portuguese, Chinese, Tamil, and any other language whose speakers do not all speak it the same way.

The number that makes the case#

In Zero-Shot Context-Aware ASR for Diverse Arabic Varieties (January 2026), Talafha, Abu Alhassan and Abdul-Mageed report zero-shot word error rates across standard and dialectal test sets. On Common Voice 15.0, reading standard Arabic, SeamlessM4T scores 11.12 and Whisper large-v3 scores 15.55. On MGB-5, reading Moroccan Arabic, the same two models score 77.43 and 79.16. Averaged over the five Casablanca dialect sets the paper uses (Algerian, Jordanian, Palestinian, Emirati and Yemeni), Whisper scores 57.46 and SeamlessM4T 59.20, so the ranking between the two models depends on which variety you test.

A word error rate near 78 means roughly four words in five are wrong. No amount of prompt engineering downstream recovers a transcript like that, and no agent built on it will pick the right tool.

Both models are excellent. Whisper’s own paper is careful about what it claims: trained on 680,000 hours of multilingual supervision, the authors write that compared to humans, “the models approach their accuracy and robustness.” Approach, on their benchmarks. The benchmarks are mostly not dialectal.

Why a language is not one thing to a model#

Transcribed audio concentrates where transcription already happened: news broadcasts, parliamentary records, subtitled film, audiobooks. Those sources are written-standard-heavy in almost every language that has a written standard distinct from ordinary speech. A model trained on that distribution learns the top of the tree well and the bottom of it poorly, and the gap is largest exactly where a consumer app lives.

Arabic makes this visible because the written standard and the spoken varieties diverge far enough to behave like separate languages in places. VoxArabica (ArabicNLP at EMNLP 2023) identifies 17 dialects alongside Modern Standard Arabic, and ships finetuned recognition models for only a few of them, with zero-shot coverage for the rest. That split is itself the finding: even a research system built specifically for the language has to treat most of its varieties as unhandled.

The measurement problem underneath the measurement#

Word error rate assumes there is one correct spelling of what was said. For a spoken variety with no settled orthography, there often is not, and two competent human transcribers will disagree on a meaningful share of tokens before any model is involved.

That has two consequences worth knowing before you read anyone’s numbers. Published dialect error rates are partly measuring transcription convention rather than recognition, so figures from different papers on the same dialect are not always comparable. And your own ground-truth set needs a written convention decided in advance and applied by everyone who transcribes for you, or your baseline will drift as you add data.

The way out, for an agent specifically, is to stop treating the transcript as the deliverable. If the agent picked the right operation with the right arguments, the turn succeeded, whatever the transcriber would have written.

What actually fixes it#

In-language data, at scale, including the varieties. At the NADI 2025 shared task, Salhab and colleagues took first place in multi-dialectal Arabic recognition by pretraining on 15,000 hours of weakly labelled speech covering both the standard variety and regional ones. Fifteen thousand hours is a rounding error next to Whisper’s 680,000, and it wins on these dialects anyway.

The practical reading for a product team is that scale on English does not transfer, and neither does a vendor’s total hours figure. What matters is hours in the varieties your users speak, and almost nobody publishes that.

The evaluation you have to run yourself#

Collect about a hundred real utterances per variety you intend to serve. Real means recorded by someone who speaks that variety, saying the things your users say, in the acoustic conditions your users are in. Transcribe them once, by hand, and treat that as ground truth.

Score per variety and never pool the results. A pooled average across five varieties hides the one that is failing, and the failing one is where your support load will come from. Publish the worst number internally, not the mean.

Include mixed-language utterances deliberately. Users switch languages mid-sentence constantly, and a pipeline that detects one language per utterance then transcribes will mangle every one of them. Code-switching happens wherever two languages share a population, from Hinglish to Spanglish, and it needs its own test cases.

Score the outcome rather than the transcript. For an agent, the question is not whether every word was right. It is whether the agent called the correct operation with the correct arguments. A transcript that reads “send five hundred” instead of “send 500” is a transcription error and a successful turn; one that reads “send” instead of “cancel” is the opposite. The evaluation set is the deliverable, and it earns its keep on every model change.

What this looks like in production#

We answer in Modern Standard Arabic rather than in the caller’s own dialect, on purpose. Understanding a dialect and speaking one are different problems, and a synthesised dialect that is slightly wrong reads as mockery in a way a neutral register does not. The recognition side takes the variety it is given; the reply side stays standard.

Right-to-left layout is a separate piece of work from language support and it is the one teams underestimate. Mirrored layout, bidirectional text with embedded Latin strings and digits, and numerals that may be rendered in either script all have to be handled by the widget set rather than patched per screen.

When a variety is genuinely thin#

Sometimes the measurement comes back and the answer is that no available model works for a group of your users. Three honest responses exist, and inventing a fourth is where products get into trouble.

Route it to a human. If the agent cannot understand a user reliably, handing them to support quickly is a better experience than three failed attempts followed by a wrong action. Make that path fast rather than buried.

Narrow the operation set for that variety. Recognition errors are much less costly when the agent is choosing between four clearly distinct operations than when it is choosing between twenty. A smaller surface converts a transcription problem into a tolerable one.

Collect data. If the variety matters commercially, a few thousand transcribed utterances from your own users is a tractable project and it is the only thing that moves the number. The NADI result above is what in-language data does at scale, and the same direction holds at small scale.

What not to do is ship it anyway and let the completion rate tell the story. Users who have a bad first experience with a voice feature do not usually give it a second one.

A checklist before you commit to a vendor#

Ask which varieties are represented in training and in what quantity, and treat a refusal to answer as an answer. Ask what the published accuracy figure was measured on, and whether you can see the set. Run your own hundred utterances per variety before signing anything. Decide which varieties you will support properly and say so publicly, because partial support that nobody documents reads to users as a broken product.

If a variety you need is genuinely thin, the honest options are to route it to a human, to constrain the agent to a smaller and more distinguishable operation set, or to collect data yourself. Well-resourced and thinly-resourced varieties behave differently enough that the decision is worth making per variety, and shipping for a variety with little training data is the case where those options stop being theoretical. The output side has the same problem in mirror image: what breaks on the way out is a different set of failures from the ones on the way in. The wider stack this fits into is covered in the complete guide to adding an agent.

Sources#

• Bashar Talafha, Amin Abu Alhassan and Muhammad Abdul-Mageed, Zero-Shot Context-Aware ASR for Diverse Arabic Varieties, January 2026. Accessed 12 September 2026.
• Alec Radford and others, Robust Speech Recognition via Large-Scale Weak Supervision, 6 December 2022. Accessed 12 September 2026.
• VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System, ArabicNLP at EMNLP 2023. Accessed 12 September 2026.
• Mahmoud Salhab and others, Munsit at NADI 2025 Shared Task 2, 12 August 2025. Accessed 12 September 2026.