Posted by John Kleeman
I shared in a previous blog article how PwC explained at the recent Questionmark Users Conference some good practices for diagnostic tests that allow people to test out of training. In this article, I’ll explain what they call their “half-time” strategy, which enables even greater time savings.
A downside of a test-out strategy is that everyone, including the people who fail, have to spend time taking the test. PwC have found that in some cases, particularly if the test is difficult to pass, having a test-out can actually increase the total time taken for pre-testing and training. Although the test may add learning value in itself, the aim of testing is to reduce time spent, not to increase it!
PwC’s “half time strategy” deals with this issue by dividing the test into two halves; if a test taker does poorly in the first half, they don’t attempt the second half as they would have no chance of passing the test. This reduces the time spent in taking the test for many people, and so improves the overall efficiency of the process.
See the diagram below for a visual representation.
Essentially, if someone is doing badly at the test, it’s best to tell them so and stop wasting their time. If the test taker is doing poorly on the first half, cut their losses right then and there.
Suppose, for example, the pass score is 80% and you need to get 14 out of 17 questions right to be able to pass the test and test out of the training. Then you might divide the test into a first half with 9 questions and a second half with 8 questions. If you don’t score at least 6 out of 9 in the first half, then even if you get all the questions in the second half right, you couldn’t pass it. So people who score less than 6 in the first half of the test needn’t take the second half.
In my previous post, I gave an example of 1,000 people who need to take compliance training and could test out by taking a 20-minute test. It takes 333 hours for people to complete the test, but in the example I gave, time was saved overall by reducing time spent training.
Using the half-time strategy, suppose that 45% of those people drop out after the first half of the test — so they only need to spend 10 minutes on the test rather than 20 minutes. Then you save an extra 75 hours of test taker time in aggregate (450 x 10 minutes). This approach also prevents learner frustration by not requiring someone to continue with a test they are going to fail.
The half-time strategy is very easy to implement in Questionmark and seems a useful technique
I hope this idea is useful for you.
You can see some other articles inspired by PwC’s practices on the Questionmark blog at Applying the principles of item and test analysis to yield better results and How many items are needed for each topic in an assessment? How PwC decides.