Text-to-speech for agents when the language is hard

Speech synthesis sounds like a solved problem if you only ever hear it read English marketing copy. The demos are excellent, the voices are warm, and nothing in them tests the parts that break.

An agent tests them constantly. It says amounts, dates, product names from another script, questions that have to sound like questions, and confirmations the user is about to act on. This post is about the stages where that goes wrong, and it uses Arabic as the worked example because Arabic breaks the two hardest ones at once. The reasoning generalises to Hebrew, Persian, Urdu, Thai, and any language whose writing records less than its speech.

The two stages that matter are not the ones being benchmarked#

The public benchmarks measure the acoustic model and the vocoder, because those are the parts a research paper can compare. StyleTTS 2 is the usual reference point, and it is worth reading the claim exactly. Li and colleagues report that the model “surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers”.

Three qualifiers are doing work in that sentence. Both datasets are English. Both are read speech rather than conversation. The judges were native English speakers. The abstract carries no mean opinion score numbers at all, so anyone quoting one from that paper has taken it from the body or from somewhere else. None of this diminishes the result; it just means the result is about a stage that was already good, in a language that was already well served.

The stages ahead of it are where an agent’s voice actually fails.

When the spelling does not determine the sound#

Arabic is normally written without short vowels. One written form carries several readings, and the reader resolves them from context so fluently that the ambiguity is invisible until you ask software to do it.

That resolution step is called diacritization, and it is a real natural-language task with its own literature and its own benchmarks. A useful calibration comes from Skiredj and Berrada’s January 2024 paper on Arabic diacritization, which reports that on the benchmark dataset from Fadel and colleagues, “GPT-4 exhibited lower performance levels, with a DER of 20% and a WER of 30%”.

A general-purpose frontier model getting one diacritic in five wrong is the number to remember. Diacritization is not something you get for free from a good model, and it is not something a synthesis vendor solves by shipping a nicer voice. If your pipeline sends undiacritized text to a synthesiser, the synthesiser is guessing, and it will guess confidently.

The practical options are to diacritize before synthesis using a model built for it, to constrain what the agent says so the ambiguous cases do not arise, or to accept a known error rate and measure it. All three are defensible. Pretending the stage does not exist is not.

This is the mirror image of the recognition problem, where the same variety differences show up on the way in. How to evaluate dialect and variety coverage covers that side, and the two are usually procured separately and then discovered to disagree.

Prosody is what makes an agent sound like an agent#

An agent’s speech is a short spoken answer that accompanies something on screen, with one job: tell the user what happened and what to look at. That is a narrower job than narration, and it wants a differently tuned voice.

Questions have to rise, and they have to rise on the right word. A generic voice reading “did you mean the Thursday appointment” flat turns a clarifying question into an announcement, and the user does not answer it.

Numbers have to be spoken as numbers. Arabic number agreement is genuinely intricate, with the form of the counted noun depending on the count, and a system that reads digits individually has produced something no speaker would say. This is a text normalisation problem, not an acoustic one, which means it is solved before the voice ever gets involved.

Confirmations need pace. The sentence before an amount should slow down and pause, because the user is being asked to make a decision and speech that arrives at the same rate as everything else does not signal that.

The reason these matter more for an agent than for an audiobook is that an audiobook’s listener has no decision to make. The prosody of a confirmation is functional.

Mixed script, and what actually breaks#

Two different problems get filed under right-to-left support and only one of them is about synthesis.

Display order is governed by Unicode Standard Annex #9, revision 51 of Unicode 17.0.0, dated 13 August 2025. The annex is explicit that most text needs no help: “in most cases, there is no need to include additional information with the text to obtain correct display ordering”, because each character carries a bidirectional type. It is equally explicit about the exceptions, noting circumstances “where an implicit bidirectional ordering is not sufficient to produce comprehensible text”, which is what the directional formatting characters exist for. Mixed Arabic and Latin with digits and punctuation is exactly that case, and a product name, an order number or a currency code dropped into an Arabic sentence is where it shows up.

Synthesis has its own version. A Latin product name inside an Arabic sentence needs the voice to switch and switch back cleanly, and many voices either mangle it or fall into spelling it out letter by letter. Test with your actual catalogue, not with a sentence someone wrote for the test.

The screenshot above shows the third problem, which is neither of those. The turn is fully Arabic, the layout is correct, and the confirm button says something in English, because the interface string was never part of the language review. The failure of a localisation effort is almost never the model.

Standard or dialect, and why the question is asked backwards#

Teams shipping to a region with a formal written standard and several spoken varieties usually frame this as a single choice: does the agent speak the standard or the local variety?

The choice is not symmetrical, and the asymmetry is worth understanding. A formal standard is what people read, what news is broadcast in, and what almost all recorded training data is in, so the voices are better and the coverage is wider. It is also not what anyone speaks at home, so an agent using it can sound institutional in a context where institutional is not what you want.

A local variety sounds natural to the people who speak it and slightly wrong to everyone else in the same market, which is the part teams underestimate. Choosing one dialect is a positioning decision made by an engineer selecting a voice, and it will be read as one.

Our own deployment uses the formal standard for Arabic, deliberately. The reasoning is that neutral and slightly formal is a safe register for a financial assistant, that it is intelligible to every speaker, and that it avoids picking one country’s variety to speak to a region. That is a defensible default rather than a universal answer. A food delivery app in one city has the opposite calculus, because there the local variety is the whole point.

What makes this decidable is that it is separable from recognition. The agent can understand every variety and answer in one, which is how most people already talk to institutions.

How to choose, without a provider table#

A per-voice comparison goes stale within a quarter, so this post no longer carries one. Four questions produce a better answer than any table would.

Does the provider expose control over pronunciation, through a lexicon, phoneme input or an equivalent? Without it you have no repair mechanism when a name is read wrong, and there will always be names read wrong.

Does it accept diacritized input and use it? Some voices ignore the marks, which makes the whole upstream stage pointless.

How does it behave on your own worst sentence? Build a test set of twenty real utterances from your product, including the longest amount, the most awkward product name, and a question. Twenty utterances take an afternoon and tell you more than every benchmark in this field.

What does it cost in latency, measured as first audio rather than as model inference? Vendor inference figures exclude the network, the server overhead and the player buffer, which together usually exceed the inference time. Where a turn’s seconds go covers how to read those numbers.

What this costs you if you skip it#

A wrong voice reads as carelessness, and the user extends that judgement to everything else the agent says, including the parts that were right.

That is the argument for treating synthesis as part of the conversation design rather than as a vendor choice made once. The conversation-design rules the voice has to serve cover the wider surface, and why some varieties of a language are better served than others covers what to expect when you move from the best-resourced variety to a neighbouring one. The voice documentation covers the synthesis path in the SDK and what it reports per turn.

Sources#

• Li, Y. A. et al., “StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models”, arXiv:2306.07691, June 2023, revised November 2023.
• Skiredj, A. and Berrada, I., “Arabic Text Diacritization In The Age Of Transfer Learning: Token Classification Is All You Need”, January 2024.
• Unicode Consortium, “Unicode Standard Annex #9: Unicode Bidirectional Algorithm”, revision 51, Unicode 17.0.0, 13 August 2025.