Welcome to this Questionmark podcast. Questionmark podcasts bring you news, ideas, and advice about assessments and learning.

Joan Phaup, Questionmark: This is Joan Phaup, from Questionmark. I’m pleased to be spending a few minutes with Dr. Will Thalheimer of Work-Learning Research. Will helps his clients build more effective learning interventions and work-learning environments. He is renowned for taking a research-based approach to his work, and bridging the gap between research and practice. He not only provides consulting workshops and learning audits. He is also now partnering with Centrax Corporation to build research-based e-learning. Our topic today is providing learners with feedback, a subject Will has researched for many years, and the focus of one of his many white papers. Welcome, Will.

Dr. Will Thalheimer, Work-Learning Research: Thanks Joan. It’s great to be here, and you know, I ought to take this opportunity to thank Questionmark for supporting my research writing efforts over the years. You guys really do a great deal in that area, and I see you as one of the enlightened and more public-spirited of the companies in our industry. So thanks so much for that.

Joan: Thank you. I’m interested in talking with you today about feedback, because although we generally agree that it’s helpful to learners, there’s a lot of confusion about using it well.

Will: Yeah. Actually a recent research to practice report that I wrote involved one of the most difficult research efforts that I’ve ever made. The research on feedback is really confusing, and there’s a lot of myths about it as well. So hopefully today, we can iron out some of that. One of the things I will tell people right up front is that there’s no easy recipes for feedback, and it’s really helpful if people have a deep understanding of how feedback works from a cognitive perspective. I really recommend that people get the paper. I think Questionmark has it available for free on your website. It’s also available on the Work-Learning Research website. And that’s the best place to go to get the in-depth information. What we’re going to do today is cover some of the highlights of that work.

Joan: Great. Thanks for that, Will. Could you expand right now on the value of feedback, and what role it plays in the learning process?

Will: Sure. One of the housekeeping items here is that when I’m talking about feedback, I’m talking about feedback in the learning process. There’s also feedback that, for example, goes on in the workplace, where managers give their direct reports feedback on their job performance. That type of feedback has some overlaps, but it’s really different. So it’s important to keep those two things distinct. You asked what role does feedback play in the learning process, and I think that’s a really good question, because feedback has to be understood in context. It’s not an isolated event. So I’m going to lay it out for you right now. Typically what happens is we give our learners some learning information. We present information to learners later. We ask them to practice. We ask them to retrieve information from memory. Only after that retrieval opportunity do we ask people, or only after that retrieval opportunity do we give people feedback. So feedback always comes after we’ve asked our learners to do something. And particularly we’ve asked them to retrieve information from memory. Now there’s one other design. So the first design is, we present information, we give them a retrieval opportunity, then we give them feedback. Another design is, we just give them a retrieval opportunity. We give them a question. We give them an exercise. We ask them to practice a skill. And then we give them feedback. So there’s no information presented in the beginning there. You could call that a pre-question first design. So given that feedback always comes after retrieval, we need to know how important retrieval is. And if you weight both of those things, feedback and retrieval, one important thing to know is that retrieval is more important than feedback. Let me say that again, (be)cause that’s really important. Retrieval is more important than feedback. So the role that feedback plays is to support that retrieval. Now one distinction that we ought to make is that sometimes when people are learning something it’s sort of early in their learning process. They’re really, they don’t have a lot of depth of knowledge. They’re beginning to understand it. So we could call that early in learning. But also there’s a time when it’s later learning. They’ve understood it, and they’re really trying to practice it so that they can remember it for later. Now early in learning, we might some different types of feedback than later in learning. So for example, the pre-question first design, you know, we’d just given them a question or given them an exercise, and then we give them feedback. In that kind of situation we’re going to want to give them extensive feedback, much more extensive than if it’s later in learning, cause later in learning they need less information. They need to really be put back on the right path. So overall, when we talk about the role of feedback, it follows retrieval, and it supports the retrieval process.

Joan: Thank you. Is feedback of any value when a person has answered a question correctly, Will?

Will: Before I answer that question, let me talk about sort of the overall principle of feedback. And I’m going to actually read from page 24 of the research report, part I of the research report, cause there’s two parts. I’m reading this because I guess I like my own prose. I think it’s kind of poetic here. So here it is. “The overarching principle is to give learners the feedback they need. Give them the feedback that will help them build appropriate mental models of the concepts to be learned. Don’t waste their time with extra tangential information. Don’t overload their limited working-memory capacity. Don’t give them partial information. Don’t provide them with verbatim repetitions of the original learning material. Don’t confuse them with diversions. Don’t exhaust them with extra obligations to attend to unnecessary feedback. Instead, give them exactly what they need to develop appropriate conceptions.” The point of this is, give them the feedback they need, but don’t give them any more. So then the question is, “Is feedback of any value when a person answers correctly?” Well, it can have value. But interestingly enough, in most cases, feedback on correct answers is almost redundant. If someone knows the information and retrieves it correctly, then if you give them feedback it’s almost redundant. Now there’s some subtleties here. First of all, sometimes people guess and get answers correct. When they guess and get answers correct, then it’s almost like, from a cognitive standpoint, they do need that feedback. The other confusion here is someone may have a certain level of confidence about the information. So they might be pretty sure, but not quite sure. So their feedback may be beneficial. Again, this goes back to the power of retrieval. Retrieval is more important than the feedback. So on correct answers, if they’ve retrieved correctly and they know it, feedback is almost redundant. It may not be necessary. And that’s important, because learners don’t want to waste their time. And if we give them feedback on information that they feel they already know, they might reject all the feedback we give them, even when they need it. So it’s important not to overload them with extra feedback that they don’t really need.

Joan: Interesting. So what is the best kind of feedback for correct answers?

Will: Well, as you just heard, there’s no simple answer to that question. It really depends on, you know, ideally we’d like to be able to know if the learner really knows the information or not. But it’s really hard to, you can’t x-ray somebody’s head when they answer a question correctly. So we have to sort of figure it out. How much feedback to give. One simple heuristic is that we give people a little bit of feedback. You know, that’s correct or that’s not correct. And then give them an option of getting additional feedback. That’s one way to handle it. We can also then think about the distinction between early in learning and later in learning. So for example if it’s later in learning, we can probably give them a really short piece of information. If it’s early in learning we might want to expound a little bit more. So be careful of the tendency to give too much feedback on correct answers.

Joan: Now the answer to this next question may not be very simple either, because this whole subject isn’t simple. But I do want to ask you, how about for incorrect answers?

Will: Well, one of the primary reasons to give feedback is to correct people’s misconceptions. So they try to retrieve something from memory and they either can’t retrieve it, or they retrieve the wrong information, or they retrieve a confused piece of information. So for incorrect answers, much more important to give feedback. How to do that? Well, again, this later in learning, earlier in learning, is important. If they’re struggling with the information, or if it’s really complex information, or if it’s really early in the learning process, we give hem more extensive feedback. Again, those, the primary principle applies. Give them just the amount of feedback they need, but no more. Now, this suggests that we need to learn about our learners a little bit. We need to have empathy with where they are in the process. Over time, as we deliver training courses, as we educate people, we learn that. But it may be something we have to learn. Now if it’s later in learning, the ideal is to give them very brief feedback. You don’t want to overload them. You don’t want to give them too much. You don’t want them to turn off your feedback. So keep it short. So what that mean is, tell them what the correct answer is. Don’t just say yes or no, or correct or incorrect, but say, correct or incorrect, and here’s the right answer. That’s later in learning. Again, later in learning should be brief. Early in learning should be more extensive.

Joan: And what about the timing of feedback? Is instant feedback the best kind of feedback?

Will: Well, this is the most complicated part of the research that I looked at. One thing we know for sure is certainly not true. We used to believe that immediate feedback is always better. But research clearly shows that that’s not true. In fact, the weight of the evidence suggests that delayed feedback is better. But the research really hasn’t puzzled this out yet. So it’s not 100% clear from the research what we should do. So we need to be a little bit, we need to show a little bit of wisdom about this. So my simple heuristic is, we can give immediate feedback, but then give people a later opportunity to retrieve similar information from memory. Now, you know one of the things that we ought, we ought to sort of step back here and say, “Why are we giving people assessments, or tests, or practice opportunities in the first place?” Are we doing it to grade them? Are we doing it to bolster their learning? Are we doing it to improve our own learning interventions? Are we doing it to prove the benefits of the particular learning intervention to some stakeholders that have an interest in whether this is working or not? The stuff I’m talking about is really, how do you improve the learning process? So for example, if you don’t care about the learning per se, you’re just trying to grade people, maybe you don’t even need to give them feedback. But if you’re really trying to improve the learning process, then feedback is vitally important. So, getting back to this instant feedback, it’s often good to try to delay feedback sometimes. So instead of giving feedback on each individual question, you might have them take five questions, and then get feedback on the first. You’d answer 1, 2, 3, 4, 5, then you go back and give them feedback on the first question, feedback on the second question, feedback on the third question, etc. Another thing you can do is delay, you know the research on the spacing effect suggests that longer intervals are better. So delaying something over a day could be better than giving people feedback now(?13:10). Sometimes, logistically, we can’t do that. If we can delay our feedback a little bit, there’s probably a benefit to that. Now if we can’t do that, again the simple heuristic is, we give them immediate feedback, and then we give them another retrieval practice opportunity. Remember that retrieval is the important thing, not the feedback. So it’s really that initial retrieval practice opportunity and then feedback, and then a subsequent retrieval practice opportunity, and even more feedback. That could be a way to overcome that, or sort of solve the problem about what to do. Another thing is, well one other thing I just do want to say is that feedback, whether it’s immediate or delayed, is valuable.

Joan: I want to wrap this up with one last question. Suppose I’m about to write a quiz. What’s the most important bit of advice you can give me about writing feedback for each of my questions?

Will: If we remember that it’s the retrieval that’s more important than the feedback, the first thing that I would recommend is to write a good question. Make sure that it’s an authentic question, that it is the kind of question that will help the learner remember and retrieve at a later time. So if you’re doing workplace learning, you want to figure out, what are sort of the cues in the environment of the person? What kind of decisions will they have to make on the job? And then write a question that sort of simulates that, if we can. In the education sphere, we want to figure out, well where— If I’m teaching history, for example, where do I want people to remember the political science stuff I’m talking about? And then, okay, so I want them to remember it when they’re thinking about who to vote for in an election, when they’re reading the newspaper. So how can I sort of simulate that kind of environment, that kind of thinking in my practice questions? So it’s not all rote memorization. One of the things we think we want to do is to just ask people lots of questions. But even better is to ask them questions in an authentic situation. I talked a lot recently about situation-based learning design, and the same principles are involved here. Set your question in the context that you want people to be able to deal with it later in their real world life. So then, once you got the right question, then make sure you, the important things around feedback are, make sure you give them feedback. Feedback’s important, because people forget information. They develop misconceptions. Or they just need more reenforcement to help them remember it better. Then consider the distinction between whether you’re teaching something that’s sort of early in learning or later in learning. If it’s earlier in learning, more extensive feedback. Later in learning, less extensive feedback. And then consider the distinction between correct feedback and incorrect feedback. Feedback much more important on incorrect answers, but on some correct answers, the ones people are guessing at, or they have low confidence in, then give them more feedback there.

Joan: Thanks very much, Will. And I do agree that people can learn a lot more from reading your paper. So I hope they’ll go and check it out on the website. Thanks again, Will.

Will: Oh, Joan, it’s my pleasure.