Shipping speech for a variety with less training data

Every speech vendor supports your language. Whether they support the variety your users actually speak is a different claim, and almost nobody publishes the number that would settle it.

This post is about measuring that gap yourself. It uses Gulf Arabic as the worked example because it is the case where the shortfall is documented in public benchmarks, and because it is genuinely hard: a widely spoken variety, economically significant, and thinly represented in the corpora everything is trained on. The method transfers to any variety in that position, from Swiss German to Nigerian English to Brazilian regional Portuguese.

What Khaleeji actually refers to#

Gulf Arabic is a grouping, not a dialect. It covers the speech of Kuwait, Bahrain, Qatar, the United Arab Emirates, Oman and the eastern and central regions of Saudi Arabia, and it varies within each of those by city, generation and community.

For a product decision the taxonomy matters less than one consequence of it. If your test set was recorded in Dubai and your users are in Riyadh, your measurement is about somebody else’s speech. The general method for evaluating dialect and variety coverage applies here in full, and this post is the harder instance of it.

The size of the gap, measured#

Casablanca is the useful benchmark because it was built specifically to test varieties that had no evaluation data, including Emirati. It is a community-collected, fully human-transcribed corpus of roughly 48 hours across eight dialects (Talafha et al., EMNLP 2024).

Running current multilingual models over it produces the spread below. These are zero-shot word error rates for whisper-large-v3 without text preprocessing, from Table 3 of that paper.

Emirati came in at 62.31, improving to 52.88 once the authors applied text normalisation. The average across all eight was 69.49, dropping to 63.00 with preprocessing. Two things are worth taking from that. The first is the spread: a single language-level claim covers varieties that differ by nearly forty points. The second is the normalisation effect, which is large enough that any comparison of two models has to state whether the same text processing was applied to both.

None of this is a criticism of the models. Whisper was trained on 680,000 hours of multilingual supervision and its authors are careful to say the models “approach” human accuracy and robustness on the benchmarks they tested (Radford et al., 6 December 2022). The benchmarks they tested did not include Emirati television dialogue.

Why the hours are not where the labels suggest#

The intuitive explanation, that less speech has been collected for Gulf Arabic, is true and incomplete. The more useful version is that the large corpora contain a dialect label without containing many dialect hours.

MGB-2 runs to 1,200 hours and is 78 percent or more Modern Standard Arabic. QASR reaches 2,000 hours with Modern Standard Arabic in the majority. Both list Gulf among their dialect labels. A team reading the corpus card could reasonably conclude that Gulf Arabic is covered, and the hours behind that label are a small fraction of a corpus dominated by a variety nobody speaks at home.

VoxArabica frames the scale of the problem from the other end: Arabic is spoken by over 450 million people across 18 varieties that the system had to handle separately, with fine-tuned models available for only a few of them and zero-shot fallbacks for the rest (Waheed et al., 17 October 2023).

Modern Standard Arabic is not a shortcut#

The tempting fix is to fine-tune on the standard variety, on the theory that it is the shared ancestor. The Casablanca evaluation tested exactly that and the result runs the other way.

On the Emirati test set, a Whisper model fine-tuned on Modern Standard Arabic scored 74.24. The same base model fine-tuned on Egyptian dialect data scored 67.45. A model fine-tuned on Moroccan scored 78.37. The authors attribute the Egyptian model’s advantage to its exposure to dialectal vocabulary and everyday conversational domains, and note that Egyptian shares more vocabulary with Emirati, Yemeni, Jordanian and Palestinian speech than Moroccan does.

That result is also the clearest argument for reading the better-resourced variety’s case alongside this one, since Egyptian is where most of the dialect hours went. The practical rule that falls out: proximity in conversational vocabulary predicts transfer better than proximity to the written standard. When you cannot get data for your variety, the next best data is a neighbouring spoken variety, not the formal register.

What good actually looks like#

It would be easy to read the numbers above as a verdict that Gulf Arabic is not shippable. The evidence says otherwise.

The NADI 2025 multi-dialect Arabic recognition task was won by a system pretrained on 15,000 hours of weakly labelled speech and then fine-tuned on filtered weak labels plus a small high-quality annotated set. It reached an average word error rate of 35.69 across the dialects, against a task baseline of 93.90, and scored 22.67 on Emirati specifically (Salhab et al., 12 August 2025).

That is a third of the zero-shot error rate on the same variety. The gap between an off-the-shelf model and a purpose-built one is the whole story, and it is why a procurement decision made on a vendor’s language list is a decision made on no information.

The evaluation, concretely#

Collect ninety minutes of real requests from real users of the variety you ship to. Not read sentences: requests, phrased the way the logs show people phrasing them.

Make sure the hard half is in there. Background noise, fast speech, two people in the room, and at least some turns where an English word appears inside an Arabic sentence, which is its own problem and one worth handling as a first-class case rather than an exception.

Have two native speakers transcribe it independently and resolve the disagreements. Their disagreement rate is itself a finding, because it bounds how good any automatic number can be.

Agree how you will normalise before you score anything. Diacritics, digit forms, hamza spelling and common orthographic variants each move the measured error rate by several points, and a comparison where one model got normalisation and the other did not is not a comparison.

Then report five numbers rather than one: overall error rate, error rate on the hard half, error rate on the fifty words your product cannot afford to get wrong, the share of turns where the agent still called the correct tool, and the share where it should have asked for a repeat and did not.

That last pair is the reason a speech metric alone under-describes an agent. A recognition error that lands on a product name is usually recoverable, because the model can still resolve the intent from context. One that lands on an amount or a recipient is not, and the way dialect variation shows up once a product is live is mostly through that asymmetry rather than through the headline rate.

Designing around the residual error#

Some error will remain no matter which model you pick, so the interface has to be built for it.

Confirm what you heard for anything consequential, on screen, in the user’s own words. A user who sees the parsed amount can correct it. A user who hears it read back once, in a synthetic voice, often cannot.

Let people repeat themselves cheaply. The recovery gesture should cost one tap and should not restart the conversation.

Keep a typed path beside the spoken one. When recognition on a variety is the weak link, the keyboard is not a failure state, it is the fallback that keeps the feature usable while the speech side improves.

And answer in Modern Standard Arabic even when the user speaks dialect, which is what we do in production. Understanding has to handle the variety; the reply is clearer, and safer to synthesise, in the standard form. The synthesis side of that decision has its own constraints.

Common questions#

Is Gulf Arabic supported by mainstream speech recognition? Supported in the sense that audio goes in and Arabic text comes out. On the Casablanca benchmark, zero-shot word error rate for Emirati speech was 62.31 with a leading multilingual model, against 48.44 for Jordanian. Support is a spectrum and the vendor list does not show where on it you are.

Why is there less data for Gulf Arabic than for Egyptian? Partly collection history, and partly that the largest Arabic corpora are majority Modern Standard Arabic with dialect labels attached to a small share of their hours. A dialect appearing in a corpus description is not evidence of many hours.

Should I fine-tune on Modern Standard Arabic if I cannot get Gulf data? The published comparison argues against it. On Emirati test data, Modern Standard fine-tuning scored 74.24 while Egyptian dialect fine-tuning scored 67.45. A neighbouring spoken variety transfers better than the written standard.

How much test data do I need to make a decision? Around ninety minutes of double-transcribed real requests is enough to separate candidate models and to find the failure cases that matter. Getting the normalisation rule agreed before scoring matters as much as the volume.

Can a purpose-built system do materially better? Yes. The NADI 2025 winning system reported 22.67 word error rate on Emirati and 35.69 averaged across dialects, against a 93.90 baseline. The distance between off-the-shelf and purpose-built is large enough to justify measuring before you choose.

Sources#

• Talafha et al., Casablanca: Data and Models for Multidialectal Arabic Speech Recognition, EMNLP 2024. Accessed 12 September 2026.
• Salhab et al., Munsit at NADI 2025 Shared Task 2: Pushing the Boundaries of Multidialectal Arabic ASR, 12 August 2025. Accessed 12 September 2026.
• Waheed et al., VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System, 17 October 2023. Accessed 12 September 2026.
• Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision, 6 December 2022. Accessed 12 September 2026.