Online proctoring can be an extremely useful tool to those managing learning programs, as it can offer both flexibility while also helping ensure test results remain secure and valid. Despite the popularity of online proctoring across many industries, the standards around it have not existed until very recently.
To that end, we sat down to speak with John Kleeman, Founder of Questionmark and Learnosity EVP, about the new ATP-NCTA Assessment Industry Standards and Best Practices for Online Observation of Tests.
- For those that don’t know John, can you give us a brief recap of your background?
I wrote the first version of the Questionmark assessment software in DOS in my back bedroom in the late 80s and then founded Questionmark commercially to bring it to market. I have been with Questionmark since then, helping to lead and grow the company as assessment technology moved on.
We delivered the first web assessment software in the 1990s and have been instrumental, along with other companies, in making technology-based testing happen. I’ve also been a director of the testing industry body, the ATP, for the last several years and was ATP Chairperson.
When Learnosity bought Questionmark a few years ago, I joined the Learnosity executive team, focusing on industry relations, legal requirements, and privacy.
- What are the newly released standards, and why is it such big news in the testing and assessments community?
Although online proctoring has been widely used for well over a decade now, no standards specifically focus on it.
Online proctoring is seen in different ways from different perspectives. For most vocational and certification test takers, online proctoring is a great boon. It means you can take a test at home or in the office and don’t have to travel to a test center. For many people, like parents or those with dependents such as elderly relatives, this is a huge benefit and time saver, allowing you to take tests at convenient times rather than needing to adhere to strict timelines. Furthermore, it also supports those with disabilities to take a test using their own equipment, including their own disability aids, all without needing to travel.
In universities and colleges, online proctoring also has similar advantages, but some stakeholders are concerned about the security and privacy aspects of proctoring because students are videoed while taking tests. Some people believe students should be trusted to take tests honestly without being proctored. Still, most colleges and universities prefer to have some evidence of the integrity of remote exams and so do use proctoring.
From all angles, then, it seems apparent that a set of standards that have gone through a rigorous process of building consensus and that set out the best practices to follow would be considered extremely valuable to many people.
- These standards have been a long time in the making, can you give us some insight into what makes the creation of these standards so challenging?
The challenge around creating a set of standards is that you have to build consensus, which has taken a long time for this particular standard. The project started in 2018 as a joint effort between the Global Testing Industry Body (ATP) and the National College Testing Association (NCTA) ). There have been three sets of Chairs, and the latest who have driven this to completion are Mike Murphy of ProctorFree and Rachel Schoenig of Cornerstone Strategies, and both deserve huge credit. Most of the major proctoring vendors have also been involved along with many universities and professional users of proctoring services.
The final document is around 100 pages long and covers everything from the different varieties of proctoring services that are available (including human-driven and automated) and different kinds of proctors. As such, there is a huge amount of detail and ground to cover.
I have been involved all the way through as a member of this workgroup. My particular focus has been on the privacy and inclusivity aspects of the standard, and also ensuring that it makes sense for the kinds of use cases that Learnosity and Questionmark customers typically face.
- Do these standards make any mention of online proctoring amid the new AI landscape?
AI is increasingly being used in proctoring and so, yes, there’s a lot of coverage of this topic within the standards. Common uses of AI in proctoring include reviewing video and audio clips to spot potentially aberrant test-taker behavior and flag anything that might be awry. There is also the potential of using AI to analyze and improve proctor performance. This all needs to be done in a fair and unbiased way, which the standard goes into at length.
The central principle of the standard is that test administration and proctoring is a critical aspect of test validity. Test validity refers to the extent to which a test accurately measures what it purports to measure. By helping encourage integrity, online proctoring can be hugely valuable in allowing stakeholders to trust the results of tests. With that in mind, validity requires that AI is not biased.
- Offering flexibility while maintaining test integrity is a constant battle for those involved in testing and education. Do you see these new standards as helping organizations find that balance more easily?
I’m pleased that there is a lot in the standard about inclusivity, and ensuring that things not relevant to the construct being measured do not influence the scores and results. The standard explores good practice in dealing with people with particular needs. As an example, it encourages the respect of religious and cultural practices, like test takers being able to wear headscarves if their religion requires them or to request a female proctor.
I do think that the standard will help set a good example for all and make it easier for organizations to adopt more human-centric ways of proctoring. Testing is a partnership between the test sponsor and test taker to allow the test taker to demonstrate skill and knowledge. However, because of a need for test security, this can sometimes be too inflexible. To be successful, testing needs to balance security, convenience, and inclusivity, and the new proctoring standard can be a real step in that direction.
- To those organizations new to online proctoring or wary of their ability to remain secure, what would you say?
The key to test security challenges is to review the risks and apply appropriate measures to counter them. Good industry practice suggests that you adopt a ‘Swiss cheese’ model of security, with multiple safeguards, including detection, prevention, deterrence, and response. Online proctoring provides an element of deterrence and an element of detection of test fraud, but even so, there needs to be a layered approach that combines approaches.
A good article to learn more about this is our Comprehensive Test Fraud Checklist.
- In terms of keeping tests and exams secure beyond just proctoring solutions, what are the key bits of advice you think organizations should follow?
First evaluate the risks to your program, which might be different from other programs, and allocate your resources in a way to minimize those risks. Proctoring is usually part of the experience for any test or exam with stakes involved, but security needs to have layers, and proctoring is only part of what is needed.
One thing that is fairly easy to do and will benefit a lot of programs is to invest a bit in deterrence. Consider ways to deter people from cheating on tests. Explain to people the rules and encourage them to follow them. Warn them that if they involve third parties (e.g. to take the test for them), they might suffer blackmail later. And make it clear to people that if there is cheating, it is likely to be caught and that there will be consequences.
Other things that are important for many programs are the security of the authoring process, some sort of protection against test takers using AI to help them answer questions (a secure browser is helpful here), and protections against proxy testers, people who try to take tests instead of the people who are supposed to.
- Lasly John, do you hope that more organizations adopt online proctoring now that there’s a best practice playbook on how to manage them?
I hope that the standards will help solidify online proctoring as a sensible, safe option for delivering tests and exams. Tests give a chance for people to reach educational opportunities, work roles, and other life chances. And online proctoring makes it easier for more people in more places to take advantage of these opportunities.
If you’re interested in the new standards, the document is free for ATP and NCTA members but can also be purchased here.