Posted By Doug Peterson
In Part 2 of this series, we looked at how to determine how many items you needed to write for each content area covered by your assessment. In this installment, we’ll take a look at a few more things that must be considered before you start writing items.
You must balance the number of items you’ve determined that you need with any time constraints imposed on the actual taking of the test. Take a look at the following table showing the average amount of time a participant spends on different question types:
It’s easy to see that it will take much longer for a participant to take an assessment containing 100 short-answer questions than one with 100 True/False questions. Therefore a time limit on the assessment might constrain what question types can be used, or conversely, the time limit may be influenced by how many questions you calculate you need and the question types you want to use. There’s no hard and fast rule here, it’s just something that needs to be considered before you go off to write a bunch of items.
The time a participant requires to complete an item is not the only thing that should be considered when determining the item format for your assessment. True/False questions might have some problems because there is a 50% chance of guessing correctly, making it hard to know if the participant knew the material or was just lucky. (I look at this in more depth in this blog article.)
Multiple Choice questions are good for recall and are easy, quick and objective to score, but many assessment developers feel that they really only test the participant’s ability to memorize. Some learning professionals are critical about incorrect information being presented, which could lead to the wrong answer being recalled at some point in the future. I discuss writing Multiple Choice question here and here.
Essay questions are good for testing higher cognitive skills such as formulating a correct answer from scratch and applying concepts to a new situation. However, they take longer for the participant to answer, and scoring takes longer and is more subjective than for other question types.
Finally, you need to decide on the presentation format, which boils down to basically two choices: paper and pencil, or computer-based (including over the Internet). Each has their pros and cons.
Paper and pencil is not susceptible to technical problems. It is comfortable for people unfamiliar with computers. However, it’s labor-intensive when it comes to distribution of the materials, grading, tracking, reporting, etc.
Computer-based assessments are faster and easier to update. Results can be provided immediately. Computer-based assessments allow for question types such as software simulations that paper and pencil can’t provide. However, not everyone is comfortable with computers or can type at a reasonable rate – you don’t want someone who knows the material to fail a test because they couldn’t answer questions fast enough using a keyboard.