Posted by John Kleeman
I attended an excellent session at the Questionmark Users Conference in Baltimore by Steve Torkel and John LoBianco of PwC and would like to share some of their ideas on building diagnostic assessments.
PwC, like many organizations, creates tests that allow participants to “test out” of training if they pass. Essentially, if you already know the material being taught, then you don’t need to spend time in the training. So as shown in the diagram below – if you pass the test, you skip training and if you don’t, you attend it.
The key advantage of this approach is that you save time when people don’t have to attend the training that they don’t need. Time is money for most organizations, and saving time is an important benefit.
Suppose, for example, you have 1,000 people who need to take some training that lasts 2 hours. This is 2,000 hours of people’s time. Now, suppose you can give a 20-minute test that 25% of people pass and therefore skip the training. The total time taken is 333 hours for the test and 1,500 hours for the training, which adds up to 1,833 hours. So having one-fourth of the test takers skip the training saves 9% of the time that would have been required for everyone to attend the training.
In addition to saving time, using diagnostic tests in this way
Some good practices that PwC shared for such building such tests are:
- Blueprint the test, ensuring that all important topics are covered in the usual way
- Use item analysis to identify and remove poorly performing items and calibrate question difficulty
- Make the test-out at least as difficult as the assessment at the end of the training course. In fact PwC makes it more difficult
- Make the test-out optional. If someone wants to skip it and just do the training, let them.
- Tell people that if they don’t know the answers to questions, they can just skip them or finish the test early – there are no consequences for doing badly on the test
- Only allow a single attempt. If someone fails the test, they must do the training
- Pilot the test items well – PwC finds it useful to pilot questions using the comments facility in Questionmark
PwC also has introduced an innovative strategy for such tests, which they call a “half-time strategy”. This makes the process more efficient by allowing weaker test takers to finish the test sooner. I’ll explain the Half-time strategy in a follow-up article soon.