Tests and exams are used for serious purposes and have significant impact on people’s lives. If they are translated wrongly, it can result in distress. As a topical illustration, poor translation of an important medical admissions test in India was the subject of a major law case ruled on by the Indian Supreme Court last week.
Because language and cultures vary, fairly translating tests and exams is hard. I recently attended a seminar organized by the OECD on translating large scale assessments which gave me a lot of insight into the test translation process. If you are interested in the OECD seminar, Steve Dept of Questionmark partner cApStAn has written a blog here, and the seminar presentations are available on the OECD website.
Here are some tips from what I’ve learned at the seminar and elsewhere on good practice in translating tests and exams.
- Put together a capable translation management team. A team approach works well when translating tests. For example a subject matter expert, a linguist/translator, a business person and a testing expert would work well together as a review and management committee.
- Think through the purpose of your translation. Experts say that achieving perfect equivalence of a test in two languages is close to impossible, so you need to define your goals. For example, are you seeking to adapt the test to measure the same thing or are you looking for a literal translation? The former may be more realistic especially if your test includes some culturally specific examples or context. Usually what you will be looking for is that the test in two languages is comparable in that a pass score in the test in either language means a similar thing for competence.
- Define a glossary for your project. If your test is on a specialist or technical subject, it will have some words specific to the content area. You can save time and increase the quality of the translation if you identify the expected translation of these words in advance. This will guide the translating team and ensure that test takers see consistent vocabulary.
- Use a competent translator (or translation company). A translator must be native in the target language but also needs current cultural knowledge, ideally from living in the target locale. A translator who is not native to the language will not be effective, and a translator who does not have knowledge of the culture may miss some references in question content (e.g. local names or slang). An ideal translator will also have subject matter knowledge and assessment knowledge.
- Export to allow a translator to use their own tools. Translators have many automated tools available to them including translation memories, glossaries and automated checking systems. For simple translation, you can translate interactively within an assessment system, but you will get more professional results if you export from your assessment management system, allow the translator to translate in their system, and then re-import (as shown in the diagram).
- Put in place a verification procedure. Translators are human and make mistakes, questions can also rely on context or knowledge that a translator may not have. A verification process will involve manual review by stakeholders looking at things like accuracy, style, country issues, culture, no clues given in choices, right choice not obviously longer than other choices and different translation word choices used in stem/choices.
- Also review by piloting and looking at item difficulty. Linguistic review is helpful but you should also look at item performance in practice. The difficulty of a translated item will vary slightly between languages. Generally small errors will be up and down and roughly cancel out. You want to catch the big errors, where ambiguity or mis-translation makes a material difference to test accuracy. You can catch some of these by running a small pilot to 50 (or even 25) participants and comparing the p-value (item difficulty or proportion who get right) in the languages. This can flag questions with significant differences in difficulty; such questions need review as they may well be badly translated.
- Consider using bilingual reviewers. If you have access to bilingual people (who speak the target and source language), it can be worth asking them to look at both versions of the questions and comment. This shouldn’t be your only verification procedure but can be very helpful and spot issues.
- Update translations as questions change. In any real world test, questions in your item bank get updated over time, and that means you need to update the translations and keep track of which ones have been updated in which languages. It can be helpful to use a translation management system, for example the one included within Questionmark OnDemand to help you manage this process, as it’s challenging and error-prone to manage manually.
- Read community guidelines. The International Test Commission have produced well-regarded guidelines on adapting/ translating tests – you can access them here. The OECD PISA guidelines, although specific to the international PISA tests, have good practice applicable to other programs. I personally like the heading to one of the sections in the PISA guidance: “Keep in mind that some respondents will misunderstand anything that can be misunderstood”!
I hope you found this post interesting – all suggestions are personal and not validated by the OECD or others. If you did find it interesting, you may also want to read my earlier blog post: Twelve tips to make questions translation ready.