Posted by John Kleeman
HSBC – a multinational banking and financial services organisation – uses Questionmark Perception worldwide for delivering computer-based, online competency tests for employees to check they understand products and regulations. Matt Bushby is an Assessment Specialist and the Global Subject Matter Expert for Questionmark Perception at HSBC.
Matt has been responsible for the transfer of paper-based testing to online assessment in the UK since 2006 and has supported and advised a dozen or so other HSBC country teams in their move from different tools and systems to a joined-up, single assessment system from Questionmark. By using assessments in this way, HSBC are able to meet strict financial regulations in each country and also ensure a premium service to customers.
I asked him to share from his experience some key tips for making online assessment effective within a large company, and here is his advice:
1. Start with “why” you are assessing. You need to have an informed conversation within the business to get discussion and buy-in. Some typical questions which can stimulate the conversation:
· How many people are likely to take the assessment?
· What are the consequences for failing?
· How high or low stakes is the assessment?
· Can people re-take the test?
· What follow-on learning will you use for people who pass the test but get some questions wrong?
2. Work out some key policy questions for the assessment:
· Should it be open-book or closed-book?
· Should everyone see the same questions or should there be random selection?
· Should there be a time limit or should it be untimed?
· Do participants have to answer all questions or can they leave some blank?
· Should the assessment give feedback on wrong answers?
3. Plan the assessment before you start writing the questions. Work out the topic granularity and structure and how many questions you need in each topic. For example, a typical assessment will call in questions from many different topics and for many assessments. HSBC typically create 3 spare questions for each question actually delivered to a test-taker.
4. Agree with the business how the structure of the assessment is going to be validated and agreed, both initially and on-going as the business shifts focus over time. Typically this would be by the business reviewing the overview blueprint for an assessment.
5. HSBC finds that multiple choice questions work well for much competency assessment, providing they are written well. It’s key to provide guidance to subject matter experts in question design. Here are some key points :
· have a consistent number of choices,
· make all incorrect choices reasonable,
· avoid negative language,
· don’t make the correct choice the longest!
6. When it meets the business need, encourage scenario questions that test application rather than factual questions. Scenarios can work well as multiple choice questions, where distractors describe common mistakes. You may need to allow longer time limits for assessments with scenario questions.
7. Be careful about using non-accessible visual elements like screenshot images in questions, as any large company will likely have visually impaired employees or others with accessibility needs that need to be catered for. It’s usually best to have one assessment that all can take, rather than having to make variants.
8. Matt finds giving generic feedback to read up on a topic or syllabus area very effective, as this means that he doesn’t need to include the original question in the feedback. This also means less risk of leakage via sharing feedback between participants.
9. For assessments that are always live (for example assessments as part of on-boarding training), keep in touch with subject matter experts and the business to ensure that these are regularly reviewed and stay up to date.
10. When introducing a new assessment, follow the approach of pilot / review / amend / launch.