Questionmark are partners of SAP, and as well as blogging on the Questionmark site, I also blog on the SAP Community Site.
I’ve recently started a series of Q&A interviews on thought leaders in the field of assessment and learning within SAP, and last week I interviewed Sue Martin, who is Global Certification Manager at SAP. I thought Questionmark blog readers might be interested in one question I asked her about how SAP ensure their questions are good quality.
John : What effort goes in at SAP to make the questions good quality and set the scoring fairly?
Sue : We follow industry standard practices to get legally defensible exams into place. In order to get more experiential subject matter expertise, we moved from instructors writing questions to much more involvement from our own consulting organization. At the same time, to get the level of consistency and quality of review, we significantly increased the number of subject matter experts (SMEs) involved.
Here is an overview of the steps we follow:
- The first step is job task analysis to identify the tasks that we ask questions on. These are validated with customer organizations, partners and with our own consulting organization.
- The next step is the design phase, where we have the item writing workshops that write and review the questions. It depends on the subject matter, but there are normally at least 6 or 7 SMEs that sit in these workshops, supported by a psychometrician.
- We then have peer review from another set of SMEs after the workshop. By SMEs I mean primarily people who are actually working in that area, so for a Professional certification that would be a Senior consultant or a Platinum consultant.
- Then another group of SMEs go through the exam as beta testers and we look at the results.
- Then a group of experts goes through the questions and do a pre-standard setting process, estimating the difficulty level of each question.
- Then we have the final standard setting meeting to get to the cut score (pass score) determination phase. And then at the end of all that, we finally have the exam!
To see the rest of the interview, including questions on how SAP are finding Questionmark effective for authoring and delivery of exams and how they are transforming their certification programme, click here.