In my first post in this series, I explained that survey respondents go through a four-step process when they answer each question: comprehend the question, retrieve/recall the information that it requires, make a judgement on the answer and then select the response. There is a risk of error at each step. I also explained the concept of “satisficing”, where participants often give a satisfactory answer rather than an optimal one – another potential source of error.
Today, I’m offering some tips for effective online attitude survey design, based on research evidence. Following these tips should help you reduce error in your attitude surveys.
Tip #1 – Avoid Agree/Disagree questions
Although these are one of the most common types of questions used in surveys, you should try to avoid questions which ask participants whether they agree with a statement.
There is an effect called acquiescence bias, where some participants are more likely to agree than disagree. It seems from the research that some participants are easily influenced and so tend to agree with things easily. This seems to apply particularly to participants who are more junior or less well educated, who may tend to think that what is asked of them might be true. For example Krosnick and Presser state that across 10 studies, 52 percent of people agreed with an assertion compared to 42 percent of those disagreeing with its opposite. If you are interested in finding more about this effect, see this 2010 paper by Saris, Revilla, Krosnick and Schaeffer.
Satisficing – where participants just try to give a good enough answer rather than their best answer – also increases the number of “agree” answers.
For example, do not ask a question like this:
My overall health is excellent. Do you:
- Strongly Agree
- Agree
- Neither Agree or Disagree
- Disagree
- Strongly Disagree
Instead re-word it to be construct specific:
How would you rate your health overall?
- Excellent
- Very good
- Good
- Fair
- Bad
- Very bad
Tip #2 – Avoid Yes/No and True/False questions
For the same reason, you should avoid Yes/No questions and True/False questions in surveys. People are more likely to answer Yes than No due to acquiescence bias.
Tip #3 – Each question should address one attitude only
Avoid double-barrelled questions that ask about more than one thing. It’s very easy to ask a question like this:
- How satisfied are you with your pay and work conditions?
However, someone might be satisfied with their pay but dissatisfied with their work conditions, or vice versa. So make it two separate questions.
Tip #4 – Minimize the difficulty of answering each question
If a question is harder to answer, it is more likely that participants will satisfice – give a good enough answer rather than the best answer. To quote Stanford Professor Jon Krosnick, “Questionnaire designers should work hard to minimize task difficulty”. For example:
- Use as few words as possible in question and responses.
- Use words that all your audience will know.
- Where possible, ask questions about the recent past not the distant past as the recent past is easier to recall.
- Decompose complex judgement tasks into simpler ones, with a single dimension to each one.
- Where possible make judgements absolute rather than relative.
- Avoid negatives. Just like in tests and exams, using negatives in your questions adds cognitive load and makes the question less likely to get an effective answer.
The less cognitive load involved in questions, the more likely you are to get accurate answers.
Tip #5 – Randomize the responses if order is not important
The order of responses can significantly influence which ones get chosen.
There is a primacy effect in surveys where participants more often choose the first response than a later one. Or if they are satisficing, they can choose the first response that seems good enough rather than the best one.
There can also be a recency effect whereby participants read through a list of choices and choose the last one they have read.
In order to avoid these effects, if your choices do not have a clear progression or some other reason for being in a particular order, randomize them. This is easy to do in Questionmark software and will remove the effect of response order on your results.