Can AI Really Understand Roman Urdu? The Language Gap We Rarely Notice

The Same Question, Three Ways

Let’s start with a little thought experiment. Imagine you want an AI assistant to explain two-factor authentication in simple words, and you ask it three times by giving these three prompts:

• English: “Explain two-factor authentication in simple words.”

• Urdu script: "دو مرحلوں والی تصدیق کو آسان الفاظ میں سمجھائیں۔"

• Roman Urdu: “Two-factor authentication ko asaan alfaaz mein samjhao.”

All three prompts have the same meaning, so you’d expect three equally good answers, right? That’s what I would have guessed too.

But researchers have been testing exactly this kind of situation, and what they found is more interesting than a simple yes or no. So here’s the big question for today: does AI really understand the way we talk, or does the quality of its answer depend a little on how we choose to write?

“Understanding” Is a Slippery Word

When an AI answers you in smooth and friendly Roman Urdu, it seems as if the AI model gets you. But behind the scenes, these models learn patterns and train them from our prompts in the form of text and then predict which words should come next. That is the distinction from how a person understands a language, so it helps to be a little careful with the word “understand.”

There are three points worth keeping in mind. A fluent response isn't necessarily the correct one, since a sentence can appear perfectly fluent and yet contain an error. An AI may be very good at one kind of task in a language while being less capable in another, so the fact that it can carry on a casual conversation in Roman Urdu does not mean that it will be just as competent when dealing with a school question or a safety issue. Lastly, an answer that sounds natural may still fail to address the point you actually made, and this kind of mistake is difficult to spot when the writing is so smooth.

Which is why it tells us very little if we merely check whether an AI can respond in Roman Urdu; the true test is its performance when the answer actually matters.

So, Why Might the Results Differ?

Before we look at the evidence, let’s talk about the possible reasons. I want to be clear that these are possible explanations and not proven facts about every AI tool out there.

• Training data. Some languages and subjects simply have more text available for training than others. Urdu has more than 230 million speakers, yet researchers still describe it as under-tested compared with English.

• Different scripts. Urdu in its own script and Urdu in English letters look completely different to a computer, even when the words are the same.

• Messy spelling. Roman Urdu has no official spelling, so a word like “acha” can show up as “achha” or “accha” depending on who is typing.

• Mixing languages. Most of us switch between Urdu and English in a single sentence without thinking about it, and that makes the meaning depend more on context.

• Local knowledge. Some questions rely on local phrases, school systems, or everyday experiences that might not be well covered in the material the AI learned from.

  

What Researchers Actually Found

I could run a few quick tests of my own and tell you what I felt, but a handful of chats can’t prove very much. So let’s look at three recent studies that tested this in a careful, controlled way. The big surprise is that the story isn’t “English good, Roman Urdu bad".

Study one: noticing when someone is struggling. A research team led by Waleed Jamil and Saima Rafiwrote 400 safety-related messages, such as someone feeling overwhelmed, being threatened, or being pressured, and then wrote each one in English, mixed Roman Urdu, and fully Roman Urdu, which gave them 1,200 test inputs. They tested seven popular models to see how often each one correctly noticed the risk. Every model did best in English, and in Roman Urdu the scores dropped by anywhere from about 4% to 23% points on average. The biggest drops showed up for quiet, indirect messages, the kind where a person doesn’t say things directly. Newer reasoning-style models handled the change better than smaller, faster ones, but none of them closed the gap completely. The authors also admit that their test set was built artificially and that the models have probably been updated since.

Study two: the plot twist. Then there’s a study called “Does Script Matter?”, which flips the expectation. Researchers asked six models to decide whether a piece of text was positive or negative, using 250 examples for each of three writing styles. Roman Urdu scored the highest, with an average accuracy of 83.9%, while mixed Urdu and English scored the lowest at 75.8%, and Urdu script landed in the middle at 77.5%. Even better, the team checked 63 of the mistakes by hand and found that almost half were not the AI’s fault at all, because the labels in the original datasets were wrong. Once that was fixed, accuracy was around 89% to 91%. So sometimes a model looks weak only because the test itself is flawed.

Study three: local knowledge. The third study, UrduMMLU, collected more than 26,000 multiple-choice questions from Urdu exam materials and question banks, and then tested 30 models using both English and Urdu prompts. The best model reached about 90%, and no other model went above 85%. What stood out was that many models lost between 25 and 40 points on Urdu-centred humanities subjects compared with science subjects, which suggests the real weak spot is local and cultural knowledge. One note of caution is that this is a preprint, which means other experts haven’t fully reviewed it yet.

So, Is Roman Urdu the Problem?

Not exactly; Roman Urdu is the problem. That's what makes this so interesting.

Roman Urdu looked weaker when the job was catching quiet signs of distress and looked strong when the job was judging positive or negative feelings, and the largest gap of all had more to do with local subject knowledge than with the prompt.

The honest answer is that it depends on the task, the model, and the topic.

We should also be fair about the limits. Each study used specific models that may already be outdated, each tested a narrow kind of task, and none of them covers every subject or every way people write.

Why You Should Care

Imagine a student asking an artificial intelligence tool to explain a difficult concept in Roman Urdu and receiving a response which at first glance seems useful, yet in fact omits a crucial point, or consider the situation of someone who is trying to understand a message from a bank or a form online and has no realization that such answers have to be verified again, and also think about a person who drafts a prompt in both ordinary Urdu and English and finds that the assistant carries out the English portion correctly but misinterprets the other part.

The difficult thing is that people tend to find a well-phrased and sure-looking response believable, and as a result, few of us take the time to question it. The safety study serves as a good example of this, since it demonstrates that the same message can be handled in different ways simply because of the way it was typed. Although this does not prove that harm is all around us, it does show why AI developers should test how people actually write.

How to Get Better Answers

The good news is that a few simple habits can help, no matter which language you prefer.

1. Say what you want and why. Tell the AI who the answer is for, like a class nine student or a shopkeeper with no technical background, so it picks the right level.

2. Explain local words. If your question depends on a local phrase or custom, add a short explanation so nothing gets lost.

3. Try again if the answer feels thin. Rewording the question, or switching the script, can show you whether the answer is stable.

4. Ask it to be honest about doubt. Something like “tell me which parts you’re sure about and which parts I should check” often gets a more careful reply.

5. Re-verify important and sensitive details. For health, money, legal, or security questions, an official source or a qualified person should have the last word.

6. Don’t trust a confident tone. Smooth wording tells you very little about whether the content is right.

You can even try a quick test yourself. Give any prompt related to everyday life, and write each one in English, Urdu script, and Roman Urdu; for every version, use a new chat, and compare these answers for accuracy, completeness, and clarity. In general, it won’t prove anything, but it will show you how reliable these AI tools are for the questions you actually ask.

And here’s something I feel strongly about: there is no need to switch to English just to get a decent answer. It is true that better prompting helps a lot, but the bigger responsibility belongs to the companies building these tools. They need to test and improve how their models handle different languages, scripts, and mixed writing.

Replying Is Not the Same as Understanding

Let’s go back to our three questions from the start. They may all have answered looking equally polished, while one is a little less accurate, a little less complete, or a little less careful than the other two. That’s the real lesson here: that AI tools are able to produce text in many languages, but it is not the same as understanding the prompt well, and things like accuracy, context, and consistency matter just as much as fluency.

So if an AI can speak our language, shouldn’t we expect it to understand the way we actually use it?