Faisal Saeed is the Founder & CEO at Promptev Inc. At the forefront of AI’s next leap.
getty
Somewhere in your company, there is probably an AI system that passed every test you gave it and is now quietly getting things wrong in another language. I say this with some confidence because I have watched it happen more than once, and because the way many companies evaluate these systems practically guarantees it.
The pattern is familiar. A team builds a document assistant or a support bot. They test it in English, it performs well and everyone signs off. Then it goes live in markets where customers write in Hindi, Arabic or Urdu, and nobody thinks to check whether the thing still works. The unspoken assumption is that language is a coat of paint. If the model reasons well in English, surely it reasons almost as well everywhere else.
It does not. The gap is far wider than most leaders expect, and the research on this point is not subtle.
What do the benchmarks actually show?
A multilingual evaluation published in 2025 tested 25 leading models across 13 languages. The best of them scored above 70% accuracy in English yet dropped to around 40% in Swahili. And that is the good end of the range. When researchers built a benchmark for Tibetan, GPT-4 fell from 68.9% to 17.5%, while Qwen-2.5-72B collapsed from 84.7% to 16.5%.
Sit with that second set of numbers for a moment. A system answering one in six questions correctly is not a weaker version of the product you approved. If a vendor pitched you software at that accuracy level, you would show them the door.
Why does this happen? Mostly because the models never saw much of these languages. Meta has disclosed that just over 5% of Llama 3's pretraining data is non-English, and that small slice has to cover more than 30 languages. There is also a cost problem hiding in the plumbing. Oxford researchers found the same text can take roughly three times as many tokens in Arabic as in English. You pay per token, so your unit economics can triple. The context window shrinks by the same amount, which means a contract that fits comfortably in English may not fit at all.
The failure is silent, and that is the real danger.
None of this would worry me if the systems failed loudly, but a model out of its depth returns fluent, confident and nicely formatted text that happens to be wrong. Latency looks fine. No errors are thrown. Every dashboard stays green.
Meanwhile, the people who could catch the problem were never invited to look. If your QA team, your test set and your acceptance criteria are all in English, you have built a review process that cannot see the one failure you most need to find. I have sat in discussions where everyone was genuinely confident, and the confidence was earned by every measure available in the discussion. The measure that mattered was not included in the discussion.
The market is also moving the wrong way for wishful thinking. The EF English Proficiency Index reported that Asia's average declined more than any other region's, driven mostly by India, with China contributing as well. In other words, your growth markets are becoming less English-speaking while your testing stays entirely English.
Here is what I would do differently.
Build your evaluation sets natively. Hire speakers of the language, use real customer documents and skip the shortcut of machine-translating your English test set. A translated test measures the translation, not the system.
Price the languages before you price the project. Arabic, Hindi and Thai deployments can cost a multiple of your English projections purely because of tokenization. That belongs in the business case, not in a post-launch surprise.
Resist the urge to filter retrieval by language. Matching queries to documents in the same language feels sensible. It is not. An executive asking in English about a contract written in Arabic is a completely normal request, and a language filter returns nothing for it. Detect the language, store it and show it to users. Never route on it.
Ask vendors for per-language accuracy. In my experience, “supports 90 languages” usually means text in 90 languages will not crash the ingest pipeline. If a vendor cannot produce accuracy figures broken down by language, that silence tells you plenty.
Finally, my favorite test costs nothing: Run a document nobody on your team can read. Thai is written without spaces between words. Urdu runs right to left. Assumptions buried deep in your pipeline surface within minutes once they meet a script that breaks them.
Measure first, and then fix.
Having spent some time dealing with data systems for markets across South Asia and the Middle East, I can tell you that assumption doesn't survive contact with real customer data. And the size of the gap surprises most technology leaders the first time they measure it. You cannot close a gap you have not measured, and you will not measure it by accident. Test in the language your customers actually use before they do it for you.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?