Posted By Doug Peterson
In May of 2013 I attended the American Society of Training and Development (ASTD) conference in Dallas, TX. While there I took in Ken Phillips’ session called “Capturing Elusive Level 3 Data: The Secrets of Survey Design.” I also picked up the book “Survey Basics” by Patricia Pulliam Phillips, Jack J. Phillips, and Bruce Aaron. (Apparently there is some sort of cosmic connection between surveys and people named “Phillips”. Who knew?) Over the course of my next few blog posts, I’d like to discuss some of the things I’ve learned about surveys.
Surveys can be accomplished in several ways:
- Focus groups
In this series, I’m going to be looking at #1 and #4. The self-administered survey is what we typically think about when we hear the word “survey” – taking an evaluation survey at the end of a training experience. Was the room temperature comfortable? Did you enjoy the training experience? Many times you hear them referred to as “smile sheets” and they relate to level 1 of the Kirkpatrick model (reaction). Questionmark excels at creating these types of surveys, and our Questionmark Live browser-based authoring tool even has a dedicated “Course Evaluation” assessment template that comes with a library of standard questions from which to select, in addition to writing questions of your own.
Surveys can also be used for Kirkpatrick level 3 evaluation – behavior. In other words, was the training applied back on the job? Many times level 3 data is derived from job statistics such as an increase in widgets produced per day or a decrease in the number of accidents reported per month. However, surveys can also be used to determine the impact of the training on job performance. Not only can the survey be taken by the learner, the survey can also take the form of an observational assessment filled out by someone else. Questionmark makes it easy to set up observational assessments – identify the observer and who they can observe, the observer logs in and specifies who he/she is observing, and the results are tied to the person being observed.
To write a good survey, it is important to understand the objectives of the survey. Define your objectives up front and then use them to drive which questions are included. If a question doesn’t pertain to one of the objectives, throw it out. The best results come from a survey that is only as long as it needs to be.
The next step is to define your target audience. The target audience of a level 1 survey is pretty obvious – it’s the people who took the training! However, level 3 surveys can be a bit trickier. Typically you would include those who participated in the training, but you may want to include others, as well. For example, if the training was around customer relations, you may want to survey some customers (internal and/or external). The learner’s peers and colleagues might be able to provide some valuable information as to how the learner is applying what was learned. The same is true about the learner’s management. In certain situations, it might also be appropriate to survey the learner’s direct reports. For example, if a manager takes leadership training, who better to survey than the people he or she is leading? The key thing is that the group being surveyed must have first-hand knowledge of the learner’s behavior.
A few more things to take into account when deciding on a target audience:
- How disruptive or costly is the data collection process? Are you asking a lot of highly paid staff to take an hour of their time to fill out a survey? Will you have to shut down the production line or take customer representatives away from their phones to fill out the survey?
- How credible do the results need to be? Learners tend to overinflate how much they use what they’ve learned, so if important decisions are being made based on the survey data
- What are the stakeholders expecting?
Whereas well-defined objectives define which questions are asked, the target audience defines how they are asked. Surveying the learner will typically involve more responses about feelings and impressions, especially in level 1 surveys. Surveying the learner’s colleagues, management, direct reports, and/or customers will involve questions more related to the learner’s observable behaviors. As this series progresses, we’ll look at writing survey questions in more depth.
Posts in this series:
- Question Basics
- More Question Basics
- Response Scales
- Finishing Up on Response Scales
- Tips for the Form of the Survey