Posted By Doug Peterson
In part 1 of this series we looked at the importance of fairness, validity and reliability in assessment items. In part 2 we looked at the different parts of an item and discussed some basic requirements for writing a good stimulus and good distractors.
Now it’s time to put all of this into practice. I’d like to present some poorly written items, understand what’s wrong with them, and look at how they could be improved. I’ll be the first to admit that these examples tend to be a little over-the-top, but I’ve never been known for my subtlety (!), and a little exaggeration helps make the problems I’m pointing out a little more clear. Let’s start with a simple Yes/No question.
Even if the stimulus of this item didn’t contain nonsense words, it would still be impossible to answer. Why? Because the stimulus basically asks two questions – should you beedle the diggle OR should you zix the frondle.
The stimulus is confusing because it is not clear and concise. Can you only take one of the two actions, and is the question asking which one? Or is it asking if you should take either of the two actions? An item like this is not fair to the test-taker because it doesn’t allow them to display their knowledge. We can fix this item by splitting it out into two questions.
- Yes or No: When loading your snarkleblaster, should you beedle the diggle?
- Yes or No: When loading your snarkleblaster, should you zix the frondle?
(And as long as we’re looking at a Yes/No question, bear in mind that a True/False or Yes/No question has a 50% chance of being answered correctly simply by guessing. It’s better to have at least 4 choices to reduce the probability of guessing the correct answer. For more thoughts on this, read my post Are True/False Questions Useless?)
Let’s take a look at another one:
You don’t have to be an expert on child-rearing to get the answer to this question. Choices a, c and d are ludicrous, and all that’s left is the correct choice.
This item is not fair to the stakeholders in the testing process because it gives away the answer and doesn’t differentiate between test-takers who have the required knowledge and those who don’t. Make sure all of the distracters are reasonable within the context of the question.
So what might be some plausible, yet still incorrect, distractors?
- Engage your child in vigorous exercise to “wear them out”.
- Raise your voice and reprimand your child if they get out of bed.
- Discuss evacuation plans in case there is a fire or tornado during the night.
We’ll continue our review of poorly written items in the next post in this series. Until then, feel free to leave your comments below.