The Pre-Hospital Emergency Care Council (PHECC) is the Irish EMS regulator, responsible for certifying all pre-hospital emergency care professionals including Emergency Medical Technicians, Paramedics and Advanced Paramedics. Volunteer organisations such as the Red Cross also use PHECC’s exams to certify staff on a voluntary basis. Prior to using Questionmark, PHECC needed to transport laptops to each exam location in order for participants to take the exams, which restricted them to administering only 3 or 4 exam periods per year. In order to grow as an organisation, PHECC needed an online assessment system that could manage their assessment process and deliver exams to any PC or laptop. After reviewing different vendor solutions, Questionmark™ Perception™ was chosen because it provided the highest level of functionality combined with value for money.  

In the past, PHECC was also restricted by the time and effort it took to author questions and complete a statistical review before they were used in a live exam. They have turned this around with the introduction of several question writing groups who author questions using Questionmark Live, a Web-based tool that has played a key role in the process by enabling questions to be authored and imported into an assessment quickly and easily.

Every time a group has finished creating 20 questions, the question set is used to create an assessment for the group to take and review. While taking the assessment, members can suggest changes in comment boxes that are included with each question. A Question Statistics Report is run on the completed assessments, and any questions with no comments go straight into a live assessment, while any questions that return with comments are sent back to the group for review. The report also flags any questions that have a high proportion of incorrect answers to help ensure that all the questions going into the live item bank are valid and reliable. Questionmark Perception has been invaluable in this process, with John Lally, PHECC’s Support Officer commenting that “what used to take me two days now takes me 20 minutes.”

Once the questions have passed the review process and are accepted into a live exam, PHECC also use Questionmark Perception’s randomisation features and reports to ensure that the questions perform well within the examination. Prior to using Questionmark they needed to spend up to four hours selecting questions at random and pulling them into the exam manually. They now use Questionmark to automatically pull questions at random from the item bank using a matrix system, with questions and answers presented in a random order. The Question Statistics Report which is completed after every exam, identifies any questions that are not performing well, and these are sent for review by a quality body.

Since introducing Questionmark, PHECC have been able to provide exams at any location with a computer laboratory. This has dramatically increased the number of exams they have been able to provide, to the extent that they are now fully booked every week, up to six months in advance.