Why some varieties of a language get recognised and others do not

Every language that is spoken across more than one country has a variety that software handles better than the rest. It is rarely the one with the most speakers and it is never the one that is linguistically simplest. It is the one that produced the most recorded, transcribable speech, and the reasons for that are historical and commercial.

Arabic is the clearest case, and Egyptian Arabic is the variety in question. Understanding why is more useful than the fact itself, because the same reasoning tells you which varieties of Spanish, Hindi, Portuguese or Chinese your product will quietly handle worse, before you have a single user complaining.

The numbers are smaller than you expect#

The Arabic MGB challenges are the standard public benchmarks for this, and comparing two of them settles the question.

Ali, Vogel and Renals describe the setup in their 2017 paper on MGB-3. MGB-2 was built on “more than 1,200 hours broadcast TV news recordings from Aljazeera Arabic TV programs”, which is the written standard read aloud by professionals. MGB-3 changed the target to dialectal speech, and the abstract is precise about what that cost: “A total of 16 hours of videos, split evenly across the different genres, were divided into adaptation, development and evaluation data sets.” Seven genres of Egyptian YouTube video, sixteen hours in total. The dialect identification task covering Egyptian, Levantine, North African, Gulf and Modern Standard Arabic ran on two hours per dialect.

Two orders of magnitude, within one language, between what people read aloud and what they actually speak. That gap is the whole story, and you get it in any language whose prestige written form dominates the broadcast media the training data came from.

What makes a variety well served#

Recorded speech that somebody already transcribed is the first input and the binding one. Egypt’s film, television and radio industries produced decades of output, and that output was exported across the region, which matters twice over. It created material, and it made the variety familiar to listeners elsewhere, which is why an annotator from another country can transcribe it.

A settled way of writing it down is the second. Dialects are often written informally, inconsistently, or not at all, and a variety that has been subtitled and printed for decades has conventions an annotation guideline can point at. Without that, two annotators produce two different transcripts of the same audio and your training data has a floor it cannot get under.

Annotators available at volume is the third, and it is the least discussed. Building a corpus needs people who speak the variety natively and can be hired and trained. Population concentration matters more than population size here.

The fourth is commercial. A vendor building one Arabic model first will build it for the largest single audience, and everything downstream inherits that decision, including which dialects arrive as a finetuned option and which arrive zero-shot. Waheed and colleagues’ VoxArabica is a good illustration of the shape: dialect identification across seventeen dialects plus the standard, with the recognition models finetuned on Modern Standard Arabic, Egyptian, Moroccan and mixed data, and the remaining dialects offered zero-shot through Whisper or MMS. Three varieties get the training; the rest get whatever generalises.

None of those four inputs is a property of the language.

Hours are not the only thing that changed#

The MGB-3 designers cut the data volume and changed what kind of speech it was. The second change is the harder one.

MGB-2 was broadcast news: professional speakers, a script, a studio, one register, clean audio. MGB-3 was seven genres of video, listed in the paper as comedy, cooking, family and kids, fashion, drama, sports, and science talks. Those are different speaking rates, different levels of formality, overlapping speakers, background music, laughter, and microphones of wildly varying quality.

That is much closer to what your users sound like, which is the point of an in-the-wild benchmark and also why the numbers on it look worse. A model that does well on read news has learned the easiest version of the problem. When a vendor quotes a figure for a language, ask which of these two kinds of speech it was measured on, because the answer changes the figure more than the model choice does.

The practical consequence is that your evaluation set has to contain the bad recordings. A test set of clean utterances recorded on a good phone in a quiet room will report a number that your production traffic will not reproduce, and the gap will be blamed on the vendor.

Register mixing is the other half#

A separate problem sits on top of the variety question, and Egyptian shows it clearly.

Speakers do not stay in one variety. A sentence can start in the local variety, take a technical term from the written standard, and borrow a product name or a whole clause from English or French. This is not sloppiness; it is how bilingual speakers normally talk, and it happens most in exactly the domains an in-app agent covers, because banking, delivery and travel vocabulary is full of borrowed terms.

A recognition system tuned to one variety handles the majority of a sentence and drops the switched span, which is frequently the part carrying the entity you needed: the merchant, the amount, the product. Losing four words out of forty can be worse than a uniformly mediocre transcript, because the four that went are the ones the tool call depended on.

The mitigation is unglamorous. Put your domain’s real vocabulary, in both scripts, into whatever customisation surface your recognition vendor offers, and measure on utterances that switch rather than on utterances that do not.

What the general models actually claim#

It is worth reading the claim in the paper everyone cites. Radford and colleagues’ Whisper paper, December 2022, trained on 680,000 hours of multilingual and multitask supervision, states that “when compared to humans, the models approach their accuracy and robustness”.

Approach. Not match, not exceed. That verb is doing precise work and it gets dropped in almost every summary of the result. On a well-served language reading clean audio, approaching human accuracy is close enough that the difference is hard to feel. On a spoken variety with sixteen hours of benchmark data and no finetuning, the gap between approaching and matching is where your product lives.

Where the gap reaches your product#

Recognition is the obvious one and it is not the only one. Variety identification affects which register the agent answers in. Synthesis inherits the same scarcity, because the voices that exist are the voices somebody had data to build, and the upstream stages that break on a hard language are worse for a variety with no standard orthography.

The item worth acting on is the last one. You cannot fix a vendor’s training data this quarter, and you can build your own evaluation set in a week. Two hundred real utterances from your own users, in the varieties they actually speak, saying the things they actually say to your product, is a better instrument than any published benchmark, because it measures the intersection of your domain and their speech. Published benchmarks measure broadcast news and YouTube comedy.

The general method for evaluating variety coverage covers how to build and score that set, which is why this post does not repeat the variety tree or a per-dialect error table. This one answers why the differences exist; that one answers what to do about them.

What this predicts elsewhere#

Ask the four questions of any language you are shipping into. Which variety produced exportable media. Which variety has a settled written form. Where can annotators be hired in numbers. Which national market does a vendor build for first.

The answers will usually converge on one variety, and that is the one your product will handle best whether or not anyone chose it. If your users are concentrated somewhere else, you have a problem that no amount of prompt engineering addresses, and the honest responses are to measure it, to answer in the shared standard rather than guessing at a variety, or to wait.

The variety adjacent to the well-served one is the interesting case, because it is close enough that models do not fail loudly and different enough that they fail often. The Gulf varieties are that case for Arabic, and they are where an evaluation set earns its cost fastest.

For the conversation design that makes an imperfect recognition rate survivable, error recovery and the rest of the craft is the place to go, and the speech settings cover what the SDK reports per turn so you can measure this on your own traffic.

Sources#

• Ali, A., Vogel, S. and Renals, S., “Speech Recognition Challenge in the Wild: Arabic MGB-3”, arXiv:1709.07276, September 2017.
• Waheed, A. et al., “VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System”, arXiv:2310.11069, October 2023.
• Radford, A. et al., “Robust Speech Recognition via Large-Scale Weak Supervision”, arXiv:2212.04356, December 2022.