The tested models gave weaker refugee-information answers in Sorani than in English
A multilingual evaluation shows why legal and protection information needs fluent expert review in the language used by the person seeking help.
Respond Crisis Translation with Taraaz
Global
20252 minutes
- Legal work and accountability
- Human rights activist
- Researcher or analyst
- Research
- Translation into and out of local languages
Evidence brief
Language choice is part of the safety design when an answer can affect legal rights or access to services.
What is known
Respond Crisis Translation and Taraaz had 12 linguists score 655 answers to questions refugees ask. The questions were posed in four languages to GPT-4o, Gemini 2.5 Flash and Mistral Small. For every tested model, Sorani answers scored worse than the English answers to the same questions.
The evaluation compared answers generated directly in each language. It was not a machine-translation test.
What is unknown
The study does not measure whether anyone acted on an answer or experienced a legal outcome. It does not establish performance for later model versions, every topic, or every Kurdish variety.
What transfers
For legal or protection information, start with an approved source and cite it in the response. Have a fluent subject-matter reviewer check meaning, eligibility rules and deadlines. If that review is unavailable, use the model only to help draft questions for a qualified adviser, not to issue the answer.
Workshop question
Which kinds of Sorani-language requests are too consequential for your organization to answer without both language and subject-matter review?
Tools the sources name
The sources name no tool for this case. We have not guessed one, and neither should anyone repeating the story.
What we do not claim
Everything above is limited by what its sources actually prove. This is the part they do not.
The study evaluates answer quality, not translation accuracy, legal outcomes or current model versions. It does not show that every Sorani answer is worse or that translating an English answer would solve the problem.