I had the good fortune of presenting a few sessions at the Questionmark 2010 Users Conference in sunny Miami a couple of weeks ago. It was a great opportunity to catch up with customers and learn about the priorities organizations are focusing on.
In one of my best practice sessions there was a great deal of interest in the topic of beta testing, so I thought I would put together a blog article on this in case others were interested.
Beta testing can be defined as gathering psychometric information regarding newly created questions in order to inform the creation of actual exams. Newly developed questions that have gone through the necessary editing and review processes are administered to representative samples of participants, either in advance of or during an actual high-stakes assessment. Psychometric information regarding the new questions is collected and used to build the actual assessments. Questions that have been beta tested are screened to ensure that they meet certain quality benchmarks (e.g., all questions fall into a certain range of difficulty, all questions have acceptable discrimination). These beta tested questions are then used to create the assessments built to specific structure criteria (e.g., there is an appropriate spread of question difficulty, a targeted mean test score is created, more questions are included on the assessment near the pass score if the assessment is criterion referenced, etc.).
A summary graphic describing the general beta testing process is included below:
There are a number of common models for beta testing questions, two of the most common are:
Want more details? Questionmark software support plan customers can learn more about beta testing from our best practice guide on this topic.