Have you ever answered a survey without thinking too hard about it, just filling in questions in ways that seem half sensible? This behavior is called satisficing – when you give responses which are adequate but not optimal. Satisficing is a big cause of error in surveys and this post explains what it is and why it happens.
These are typical satisficing behaviors:
- selecting the first response alternative that seems reasonable
- agreeing with any statement that asks for agree/disagree answers
- endorsing the status quo and not thinking through questions inviting change
- in a matrix question, picking the same response for all parts of the matrix
- responding “don’t know”
- mentally coin flipping to answer a question
- leaving questions unanswered
How prevalent is it?
Very few of us satisfice when taking a test. We usually try hard to give the best answers we can. But unfortunately for survey authors, it’s very common in surveys to answer half-heartedly, and satisficing is one of the common causes of survey errors.
For instance, a Harvard University study looked at a university survey with 250 items. Students were given a $15 cash incentive to complete it:
- Eighty-one percent of participants satisficed at least in part.
- Thirty-six percent rushed through parts of the survey too fast to be giving optimal answers.
- The amount of satisficing increased later in the survey.
- Satisficing impacted the validity and reliability of the survey and of any correlations made.
It is likely that for many surveys, satisficing plays an important part in the quality of the data.
How does it look like?
There are a few tricks to help identify satisficing behavior, but the first thing to look for when examining the data is straight-lining on grid questions. According to How to Spot a Fake, an article based on the Practices that minimize online panelist satisficing behavior by Shawna Fisher, “an instance or two may be valid, but often, straight-lining is a red flag that indicates a respondent is satisficing.” See the illustration for a visual.
Why does it happen?
Research suggests that there are four reasons participants typically satisfice:
1. Participant motivation. Survey participants are often asked to spend time and effort on a survey without much apparent reward or benefit. One of the biggest contributors to satisficing is lack of motivation to answer well.
2. Survey difficulty. The harder a survey is to answer and the more mental energy that needs to go into thinking about the best answers, the more likely participants are to give up and choose an easy way through.
3. Participant ability. Those who find the questions difficult, either because they are less able, or because they have not had a chance to consider the issues being asked in other contexts are more likely to satisfice.
4. Participant fatigue. The longer a survey is, the more likely the participant is to give up and start satisficing.
So how can we reduce satisficing? The answer is to address these reasons in our survey design. I’ll suggest some ways of doing this in a follow-up post.
I hope thinking about satisficing might give you better survey results with your Questionmark surveys!