The Conferences
The last two weeks have seen two major assessment conferences occur. I was fortunate to attend both E-ATP (the European Association of Test Publishers conference and security summit) in London and then the following week the I.C.E. Exchange (Institute of Credentialing Excellence) in Savannah, Georgia.
Both conferences had very full programs (I.C.E. even had sessions starting at 7:30 in the morning!) The I.C.E. Exchange was bigger at around 600 people and slightly more formal and focused on certifications and credentialing (including assessments but also governance and credentialing). By comparison, ATP was smaller, with around 300 people and so intimate and friendly with a focus on how to manage and deliver assessments of different kinds.
Six days of sessions, speaking and learning, and networking has been fantastic but also mind-blowing. Two themes were strong at both conferences – test fraud and DEI/fairness.
Combatting cheating and other kinds of test fraud is critical for the continued validity of most assessment and certification programs. Equally, there is a huge realization that testing and certification organizations can improve how they handle diversity, equity and inclusivity (DEI).
Below are some of the things I learned about these two themes.
The Themes
Test Fraud
The two hot topics surrounding test fraud are content theft, where test content is stolen and used to help future test-takers, and proxy testing, where someone takes a test for someone else.
A worrying trend on the topic of proxy testing is remote access proxy testers. This is where the test-taker appears to take the exam, but in fact, the computer mouse and keyboard are controlled by someone else. Steve Addicott of Caveon and I facilitated a coffee conversation at E-ATP looking at how to deal with such threats. One way is to remind candidates that if they do this, they are working with criminals, often international criminals, and they risk being extorted or blackmailed later on if they use such services.
On content theft, there is increasing realization that with it being so easy to capture question content (hidden cameras, screen grabbers, etc.), a good solution, and perhaps the best viable, long-term approach, is to have a very large item bank or to scramble/randomize questions. Some organizations use AI techniques. For example, I spoke with John Weiner of PSI and Alina von Davier of Duolingo on the Promise and Peril of AI at E-ATP. Alina shared that at Duolingo, they use AI to help generate psychometrically strong questions for the Duolingo English Test. Many other organizations are using randomization or algorithmic questions to vary tests between test takers.
A good way of thinking about how to prevent test fraud is set out in the forthcoming ATP Technology Based Assessment Guidelines (and also used in Questionmark’s comprehensive test fraud checklist). This suggests three general approaches to dealing with test fraud:
- Deter test-takers from considering cheating (e.g., warn them of the consequences)
- Prevent test-takers from doing it (e.g., by technical measures)
- Detect & respond to test fraud (e.g., by statistical analysis and sanctions for those caught)
I think we will see the industry coalesce on these categories of Deter, Prevent and Detect & Respond to work together on combatting test fraud in the future.
Diversity, Equity and Inclusion
There were several sessions at both conferences on how to make testing fairer and to ensure diversity and inclusivity in certification programs. One good piece of advice is to look at people who start certification or assessment programs but don’t finish or who fail. You can take these people and survey them to try to identify challenges and barriers.
There is also an increasing understanding that there is a challenge for non-native language speakers. For example, over 20% of the US population speak a language other than English at home, and one has to work out whether to translate tests or provide other routes to make exams fair for them. Steve Dept of cApStAn and I presented a talk on how fair is it to present exams only in English, (something which tools like Questionmark’s new Instant Translate feature are built to help address.)
A key message is that equity is more than just the assessment event, it’s the whole life cycle. You have to think about equity in the way that people learn about exams and certification programs and prepare for them. People from less advantaged groups may not have as many contacts or friends who’ve been successful in exams and may need mentoring or other ways to encourage them to apply.
One speaker at E-ATP even reminded us that we depend very much on society. In educational testing, it doesn’t matter how well you write your tests or set up your technology if the children taking the test haven’t been looked after well enough to have had breakfast.
The keynote speaker at the I.C.E. Exchange, Shawn Kanungo, suggested that assessment and certification are becoming even more important. In today’s world, where anyone can share opinions, it is hard to know who is an expert or who just looks like one. Credible credentials and certifications are important to identify genuine experts one can trust.
I’d encourage everyone who works in assessment to attend conferences if you can. They are excellent places to learn best practices and network, and both ATP and I.C.E. run good events. I’d encourage you to check out ATP Innovations in Dallas, Texas next March or else E-ATP in Vienna next September, or the I.C.E. conference in Colorado Springs next October.
I hope to see some of you there!