John Kleeman is joined by assessment luminary, Professor Phillip Dawson, the Associate Director of the Centre for Research in Assessment and Digital Learning (CRADLE) at Australia’s Deakin University. With an academic background in education, artificial intelligence and cybersecurity, he leads CRADLE’s work on cheating, academic integrity and assessment security. This work spans hacking and cheating on online exams, training academics to detect contract cheating, student use of study drugs, the effectiveness of legislation in stopping cheating and the evaluation of new assessment security technologies. Phillip shares his thoughts on constructing an assessment that gives feedback, reducing test cheating and the privacy concerns associated with remote proctoring.
Full Transcript:
John Kleeman:
Hello everyone and welcome to Unlocking the Potential of Assessments, the show that delves into creating, delivering, and reporting on valid and reliable assessments. In each episode, we chat with assessment luminaries, influencers, subject matter experts and customers to discover and examine the latest and best practice guidance for all things assessment. I’m your host, John Kleeman, founder of Questionmark and EVP of industry relations and business development at Learnosity, the assessment technology company.
Today we’re really pleased to welcome Professor Phillip Dawson who’s the associate director of the Centre for Research in Assessment and Digital Learning, CRADLE, at Deakin University in Australia. Phill has degrees in education, artificial intelligence and cybersecurity, and he leads CRADLE’s work on cheating, academic integrity and assessment security.
This work spans hacking and cheating on online exams, training academics to detect contract cheating, student use of study drugs, the effectiveness of legislation in stopping cheating and the evaluation of new assessment security technologies.
He’s written numerous books and Phill’s work on cheating as part of his broader research into assessment, which includes work on assessment design and feedback. And in his spare time, Phill performs improv comedy and produces the academic themed comedy show The Peer Review. Welcome, Phillip. I expect a joke every second.
Professor Phillip Dawson:
Thanks so much, John. We will see how we go there, but it’s a joy to be here.
John Kleeman:
It’s very good to have you here. So the question I tend to ask everybody is, how did you get into assessment?
Professor Phillip Dawson:
So I guess I was in an academic development role at a large university and I followed whatever the needs of the organization were and gee, there was some issues with assessment. So I was tasked with going out there and learning more about assessment. I wrote a research grant. I thought I better do this, I better do it about assessment and I chucked a bunch of assessment luminaries on there. And I guess the rest is history.
Assessment appealed to me as well as a topic area because there’s this great quote by Paul Ramsden that says, from the student’s perspective the assessment defines the actual curriculum. And that was my experience as an educator and as a student.
John Kleeman:
So do you think it’s true that the assessment does define the curriculum?
Professor Phillip Dawson:
Yeah, I think it is. And I think we see this particularly in online learning these days where there are a lot of time poor, quite strategic learners who do a working backwards approach from the assessment to guide their learning. I don’t think we have this time of students luxuriating on campus and just chilling out and sitting around and learning stuff so much anymore, at least not in the contexts I’m in.
John Kleeman:
So talk me through what you did, how you got to where you are, what your career journey has been like.
Professor Phillip Dawson:
Yeah, sure. So, I initially trained in computer science. I thought I was going to be a computer scientist. Started a PhD in robotics. It was fascinating, but alongside that, I’d always been interested in learning and teaching. I started teaching in the dotcom boom at the age of 20 because no one else was available to teach it. So they just got me, a later year undergrad to teach stuff.
I did a PhD in education. Ended up moving from roles that were quite teaching intensive over to academic integrity roles. I did a collaboration with David Boud, who’s a real leading assessment researcher. He’s the director of the center I’m in now. And gee, assessment just seemed like the most powerful thing I could be studying, the thing with the most leverage to change and also the thing that both helped and hurt people the most.
So, I got obsessed with that and then Dave started the research center at the university I’m at, which is Deakin University and I applied for the associate director job. So I’m Dave’s second in charge at this center and I’ve just been obsessed with assessment ever since.
John Kleeman:
That’s great. Well, you’re a perfect person to have on the podcast as we’re very assessment here. So tell me about the kinds of things that you and CRADLE research into.
Professor Phillip Dawson:
So I guess in some ways it’s what it says on the tin, assessment and digital learning, but that broadens out to quite a large set of things. We’re largely in higher education, but we also do work with other things, post-secondary things like collaborations with medical colleges.
We have a real strength in work around feedback. So some of the big shifts that have happened in the feedback literature over the past decade have come out of CRADLE or we’ve been in the room where they’ve happened. And within that a lot of the works around how could we equip students so they’re capable to make the most of feedback.
We’ve got a line of work around the digital, what students’ experience is in this sort of digital environment, not just an ed tech focus, but more of a broad, almost sociological focus of what’s the broad student life in this digital world.
And then I run a program of work around what I call assessment security, which I distinguish from academic integrity, which is more of a positive, educative approach. I am interested in how can we assure the security of our exams, tests, assignments, all that stuff.
John Kleeman:
So, look, I would love to ask you more about feedback and more about assessment security, but first of all, tell me about the comedy.
Professor Phillip Dawson:
Oh yes, yes. So about five years ago I got into improv comedy, which is a style of comedy where we make it all up on the spot. I did it because academia’s pretty serious and I wanted a bit of something different. So I sought out classes, took classes in it and gradually started getting into performing.
I’ve got a show on tomorrow night and the next night, which is to show the frequency that I do this, not to encourage people to come to these particular shows because your audience is all around the world. But the thing I’m most proud of is this show, The Peer Review where we get an academic in to talk about their research and then we do comedy about that. And if you have any audience who’s in Melbourne, Australia, who wants to find out about it, they can Google it and they’ll find us.
John Kleeman:
Well, next time I come to Melbourne, I must check you out, or perhaps you’ll do a guest show when you visit Europe. So, tell us about feedback. If you’re constructing an assessment that gives feedback, what are the key things that people should think about?
Professor Phillip Dawson:
Okay, so often with feedback we think how can I provide the best comments to students. And I really respect and admire that motivation, but what I challenge people to do is shift from how can I give really good comments to how can we make this a really effective feedback situation.
So yes, some of that’s going to be the comments that you the educator provide to students, but some of it’s going to be the context that you set up for students to make use of those comments within. A lot of feedback comments get end loaded right at the end of a sequence of tasks where someone doesn’t really have a use for them. No matter how great your comments are, if they’re at the final terminal point, that’s one of the least effective places for them.
So thinking about how can we build a situation where people can use the comments because ultimately no matter how great comments are, if they’re not used by the person, we’ve just wasted everyone’s time. I mean, back in the day I used to have a box with all of these uncollected assignments that I’ve written all these lovely comments on. We would say now that that’s a feedback graveyard, there’s nothing useful in there.
So really thinking about that, thinking about the capability of students as well. So how can we build the capability so that they can make best use of it? We might want to consider talking about how we make use of comments. A couple of my colleagues, Margaret Bearman, Liz Molloy have called this intellectual streaking, which they now call intellectual candor, because I think streaking was a bit too risqué outside the Australian context.
But this idea that we should lay bare our processes with things like feedback. What do we do when we get really hard feedback? And I guess the other thing is maybe think about the instances where we’ve said we don’t have to do feedback and really challenge that. So on say exams, sometimes there’s an opinion that exams are exempt from feedback. I’d like to really challenge that.
John Kleeman:
Yeah, certainly. I know a lot of certification exams don’t give feedback partly for exam security reasons. What about more automated feedback, multiple choice or other objective questions? What’s your guidance there?
Professor Phillip Dawson:
So, there’s this fascinating result from the multiple-choice literature about, I think they call it the memorial effects of multiple-choice testing. That’s a paper title where they look at what do people learn from multiple choice, and people can learn false facts from multiple choice. When you select the distractor answer, you could walk away from that test believing it as fact and that’s dangerous. But there’s also research that shows that immediate feedback completely counteracts that effect. So I would say we should probably consider feedback in those contexts if we can.
Now I’m not saying that this sort of learning false facts means multiple choice is bad. The testing effect was still found to be more powerful than the false fact effect, the testing effect being people learning from being tested, that testing is a powerful thing for learning.
John Kleeman:
Which I think is based around a retrieval practice that by giving people a chance to retrieve things, that reinforces memory. Is that right?
Professor Phillip Dawson:
Yeah, very much. And I’d still definitely support that, but if we can also provide some immediate feedback, we can really improve that learning. And it could be all sorts of forms of feedback. It doesn’t have to be individual educators providing it to students. As you said, automated feedback can be really powerful.
John Kleeman:
And is it best to give feedback immediately or later or both or?
Professor Phillip Dawson:
So there’s two things we want to consider with timeliness of feedback. One of them is how close it is to the performance that the student has done, so in this case the test. We want things to be close to that. And the other thing we want to consider is what follow-up task is the student going to do where they’re going to make use of that feedback. And we want to give people enough time so they can really make use of that. So if we have a series of tests and they’re spaced out, the more time we give someone prior to the next test, the better.
Something I’d really like to explore that I think there’s a real gap in the market around is the representation of past feedback to students at the point where it’s going to be most useful for them. So you did this well in this test and you’ve got another test three months later, how about there’s a way that in the period that we think you’re going to be studying, say a couple of weeks before this second test, we automatically send you the feedback that’s going to be most useful to you and say hey, this is what you might need to focus on.
John Kleeman:
So just in time feedback, a bit like they’re talking about just in time learning in the corporate world.
Professor Phillip Dawson:
Yeah, absolutely.
John Kleeman:
So if people want to learn more about feedback, any resources you can suggest that would be good?
Professor Phillip Dawson:
Yeah. So, we did a large national project called Feedback for Learning. The resources for that project are on feedbackforlearning.org and that’s got a lot about feedback across different disciplinary contexts. A lot of it’s about feedback in very large scale contexts. So in Australian universities we have classes of a thousand students or more. So it’s that sort of scale, things that could actually work at scale.
John Kleeman:
So that’s really interesting on feedback as I know a lot of people put a lot of time in feedback into assessments and I think there’s some really interesting insights there.
Let’s change the subject completely and go on to assessment security and let’s first of all here, because I think you’ve just published a book on assessment security, which in fact I’ve just ordered but I haven’t read yet.
Professor Phillip Dawson:
So I published a book Defending Assessment Security in a Digital World. It’s kind of a funny story. I had been really interested in writing a book on that and in the Australian summer of 2019 to 2020, I just worked for seven weeks and hammered this book out. I’d been thinking about it for years.
And then there’s this COVID-19 thing coming, but I don’t even mention that in the book. And then bang, we’re all in lockdown, especially in Melbourne where we had one of the longest lockdowns in the world and we’re all doing online assessment. So all of this sort of body of work that I’d pieced together suddenly was really important.
It’s a book that covers what can we do to secure assessment in a world where we’re not physically there with students, what seems to work, what seems to not work. How could we draw in some learnings from other disciplines, like how does the online gambling world deal with cheating? How does online gaming deal with cheating? What’s cybersecurity think? What’s artificial intelligence think about this? So, I just tried to piece together many different disciplines and come up with some practical steps that people could do to improve their assessment security.
John Kleeman:
So what would be your advice to people trying to reduce cheating at tests?
Professor Phillip Dawson:
There’s a few steps. So one of them is to really think about the learning outcomes that are being assessed. And if you could avoid assessing lower level learning outcomes, try and do so because lower level learning outcomes are so much easier to cheat in tests on.
I can cheat your lower level learning outcome exam potentially with a Post-It note on my screen that various tools won’t find, whereas higher level learning outcomes are much more challenging. I need to use more sophisticated cheating approaches to try to cheat in those. So that’s one step.
John Kleeman:
And just to check, clarify that we’re talking about not testing knowledge, testing it beyond recall, higher levels of the Bloom’s taxonomy?
Professor Phillip Dawson:
Yeah, absolutely. So in your various learning outcomes taxonomies at the bottom level, we’ve got basically your ability to memorize and recall different things. And I’m not saying that’s not important. I’m in the higher education space and we have classes like anatomy and physiology where it’s really important that you can recall the names of the bones and muscles, et cetera, in the body. And I don’t want to go to my doctor and they have to do, the hip bone’s connected to the leg bone or whatever.
So I do respect the need for automaticity around lower level knowledge, but we have to understand that it’s really hard to assess that in a secure way. So if we can jettison those in circumstances where they don’t matter so much and go higher level, there’s just fewer ways to cheat on those higher level outcomes. So that’s one thing.
And I’d say, this is going to be super obvious to a lot of your listeners, but it’s funny, it’s not to everyone, don’t do high stakes, unsupervised online exams. I really think it’s one of the worst practices that we have. In some higher ed contexts, up to 60% of the marks in a given course subject unit can be unsupervised multiple choice and I just think that’s horrendous. We know that students collaborate and collude and all sorts of other things on those.
Now I say high stakes because it can be really effective to use unsupervised multiple choice as a learning aid. Have it low stakes, have it something that’s part of a sequence of tasks. That’s great, but as a high stakes thing, it’s really questionable.
A third thing that I want to mention is really zooming out from the level of an individual test to if you have a sequence of tests or if you have a whole large course or something, try to view those all together as a program and think for whatever award I give people at the end of this, how am I sure across this sequence that they’ve done it themselves in the circumstances we’ve asked them to do?
In the higher ed space, I talk with people about, think about the security of the degree, don’t think about the security of just the individual task. And there might be some moments where it really matters to secure the task and really invest heavily there.
There’s some approaches that seem pretty good for those high stakes moments. Interactive oral assessment is one that we encourage in higher every day. There’s work around remote proctoring in those contexts. Remote proctoring’s imperfect, but there’s good evidence that it reduces rates of cheating. So, it has a deterrent effect. People do tend to score lower in remote proctored versus unsupervised tasks. There’s at least a dozen studies on that.
It’s not perfect at detecting cheating. I can guarantee you I could cheat in any given remote proctored exam if you give me enough time to prepare and think about it, but overall, it’s going to be pretty good. And if I may, one last thing. I said my last one was the last one.
John Kleeman:
No, no, there’s lots of interest in this, so you can speak more. Carry on.
Professor Phillip Dawson:
Okay. The really big thing to think about is how we layer many types of interventions together in a sort of Swiss cheese approach. There’s a researcher called Reason who proposed this long time ago that when we’re dealing with trying to reduce risk or problems like this rather than looking for a perfect intervention, we might get many imperfect interventions and layer them like Swiss cheese where the holes don’t all line up and things might get past one or two of these interventions, but they don’t get past everything.
In the pandemic we’ve seen this with imperfect vaccines and imperfect masks, imperfect social distancing and a whole bunch of other interventions that we as a society decided to layer together because they do a better job than any single intervention.
And Kiata Rundle has proposed that we should do this with cheating and academic integrity and assessment security, that we might want to layer imperfect approaches. And that’s how something like remote proctoring, which is imperfect, can still have a role to play as a layer of Swiss cheese in an overall model.
John Kleeman:
So essentially I think what you’re suggesting is don’t try and have a multiple assessment event so that each individual assessment event isn’t so crucial, and then for all your assessment events have a multi-layered Swiss cheese type model of different security interventions or approaches.
Professor Phillip Dawson:
Yeah, absolutely. And in different contexts there’s going to be an appetite for different sorts of approaches. I know that the thing I said about cutting out the lower level learning outcomes, that’s not going to appeal to people in some contexts, but in that context you might decide that, oh, we can’t invest in interactive oral assessment for everybody, but we’re going to do a random audit of people.
So we’re going to pick a name out of a hat and we’re going to have an interactive conversation with that person and all the students are going to know that that’s the thing that happens like with tax fraud where we know that they don’t audit everyone, but there’s a chance I might get audited so I don’t defraud my tax office. So, things like that. We might decide, oh, our Swiss cheese approach is going to consist of these layers which are different to the layers used in some other context.
John Kleeman:
So let’s talk a little bit more about remote proctoring because in the corporate and high stakes certification world, it’s pretty routine because everybody’s used to being on Zoom calls and things like that all the time, and so just taking your test in front of it. But in the higher education university world, it does seem to be a little bit more controversial with some people saying it’s terrible and others saying that it just is common sense. What about the privacy aspects of videoing people who might be at home or other things?
Professor Phillip Dawson:
So, I have the joy that whenever I give a talk about remote proctoring, I get the remote proctoring haters telling me that I’m biased, pro remote proctoring and I’ve had some pretty nasty tweets and emails. And then I also get nasty tweets and emails from the people who are pro remote proctoring who tell me you hate it and you’re being unreasonable and all that, and that makes me think I’m probably on the right track.
So I developed some guidelines for our higher education regulator, TEQSA, Tertiary Education Quality Standards Agency in Australia around the use of remote proctoring and there’s a lot of considerations. One thing that I learned in the process of writing that was that a lot of what we think are problems with remote proctoring are problems with poor use of remote proctoring.
There was famously pictures of jugs of urine from students sitting a remote proctored exam, and this was used to claim that remote proctoring is terrible because it doesn’t give people bathroom breaks. No, that’s a choice. Not giving people bathroom breaks in a remote proctored exam is a designed choice that’s been made there. There’s stuff around-
John Kleeman:
A very odd design choice in my view, but yeah.
Professor Phillip Dawson:
Yeah, I agree with you. But it’s not an inherent fault with remote proctoring. The same could be said if we didn’t let students have bathroom breaks in face to face exams. So you got that, you’ve got things about remote proctoring being biased against people with dark skin and certainly there have been instances where that’s happened. It’s not an inherent thing in it that’s mandatorily a part of all of it. That’s a bad implementation of it that I agree is a racist one that comes with all these problems, but it’s also not inherent to remote proctoring.
So I think we do need to shake out what’s poor practice that we would all agree is terrible abhorrent stuff to do and then what stuff that’s inherent within remote proctoring. So certainly the privacy concern to me is a real one that no matter what your implementation of it, we are doing a tradeoff of privacy for assessment security.
Now I’ve done interviews with students and we’ve done surveys and all that sort of stuff and it really wasn’t an issue for most students, the privacy part. The tradeoff of being at home in your pajamas for the privacy invasion was very much a trade off a lot of these students were happy to do. Our students hate finding parking on campus and all that set of things.
I think we do need to ask about the normalization of surveillance and if this is something that we’re happy to do. I think we should have frank conversations with our students about why we do this and why we think it’s an acceptable trade off in this circumstance. But we’re not endorsing mass surveillance as a general concept and I think we need to have a conversation with people about that.
And then I think we need to have conversations about the efficacy of remote proctoring. I have gone out and asked many vendors of remote proctoring tools if they’ll let me do a study where I try to cheat in their exams and publish about that. There’s been no appetite for that sort of study.
And look, I understand it, no one wants a bad news bit of research, but the danger to them is that there now are studies coming out where people haven’t made that request, they’ve just gone and done the study without a partnership and they’ve just gone and published these results.
And I think we need better partnerships between researchers in cheating and proctoring vendors about how can we do this in partnership in a responsible way. So the researchers let the proctoring vendors know when there’s problems and we work together to address them because I think we’re at the moment veering off into another direction where researchers are just trashing the space and publishing all this stuff. I prefer what the security world calls a responsible disclosure model.
John Kleeman:
Well, that sounds interesting and maybe we could talk about that outside that. But I think the Swiss cheese model seems very good because clearly proctoring on its own is not going to stop all cheating, but it will reduce it, and the combination of that and other things.
Let’s move on to AI and the future and things. So I guess it doesn’t apply so much to more objective or multiple choice type questions, but essays, I know there are tools out there that can write essays for you. How’s that impacting the world of higher education?
Professor Phillip Dawson:
So I really think in the next few months to a year we’re going to see higher education need to change its assessment designs radically because the tools that I use can produce not great work, but they can produce passable work. And we have a real standards problem at where that pass level is in higher education. We’ve got to remember the pass level is where we give you a degree. That’s a big deal. And these AI tools can produce that sort of pass level in a variety of different assessment types.
They are much more sophisticated than we give them credit for. It’s not just standard text stuff. You can get AI to write you computer code. And the really fascinating thing is the AI, something like GPT-3, which is one of the models was trained on just text from the internet, but it learned how to write computer software because there’s enough text on the internet that tells you how to write computer software that you can ask it to do that. I could ask the tool that I use to write me a limerick on a topic, and it gives me a pretty decent limerick. They’re sophisticated tools that know quite a lot.
Now there’s one line of thinking that says let’s ban them, but I don’t think we want to go there because these tools are part of authentic professional practice. The other line of thinking is let’s try and accommodate them into our assessment and I think that’s where we want to go.
We don’t want to let students use them all the time because we do need students to learn how to do fundamental things in the absence of those supports, but we also don’t want to tell students all the time you can’t use these tools because it’s like my year eight math teacher telling me, you can’t use a calculator because you won’t have it in your pocket with you all the time. And I do now very much have it in my pocket and on my wrist.
John Kleeman:
So the future is going to be everybody’s going to have AI tools and AI helpers and things and assessment needs to take that into account is what I think you’re saying.
Professor Phillip Dawson:
Yeah, absolutely. And I think we need to think of this as an increase in human capability rather than a threat to integrity. So the things that final year university students will be able to do in five years’ time will be much greater and grander than what they were able to do just a few months ago because they’ll have these tools and it will increase their capability.
John Kleeman:
And so how does the future of assessment look?
Professor Phillip Dawson:
Oh, great question. We try and address this in one of our books, Re-Imagining University Assessment in a Digital World. And there’s a great chapter in there by Margaret Bearman and Rose Luckin that talks about, so if AI can do everything or an increasing amount of things, what do we need to do in assessment? And they say we need to look at what’s fundamentally human, what do humans still need to be able to do in this world and how do we map that back to assessment.
And a key capability they talk about is evaluative judgment, which is your understanding of what quality looks like and your ability to make decisions about quality in your own work and the work of others. And they argue that this is going to remain quite a human capability that students will use these tools, but they’ll need to make judgments about the quality of them.
My friends who are copywriters whose job has changed from writing copy to using AI tools that write copy, their job relies on their expertise with text and writing, their understanding of what good is and their ability to see good in the outputs of these AI systems. And I think that’s going to be an increasingly important capability in this new world.
John Kleeman:
Very interesting. Thank you. And just remind us of your book title again.
Professor Phillip Dawson:
The book is Re-imagining University Assessment in a Digital World. That’s the second book. I’ve plugged two books.
John Kleeman:
Oh, you plugged two books. So, the first book was Defending Assessment Security in a Digital World. Oh well, it’s late at night here as we’re recording this because Phillip’s in Melbourne and I’m in London. So the first book was Defending Assessment Security in a Digital World. And the second book is?
Professor Phillip Dawson:
Re-imagining University Assessment in a Digital World. The digital world part is quite funny. I didn’t think of it when I was naming the assessment security book, that we’d used that part of the title already, but it fits. It’s about the same idea.
John Kleeman:
Okay. Okay, well I must get that one as well. Thank you, Phillip. It’s been really, really interesting talking to you today. And thank you to the audience as well. Thank you for listening and we really appreciate your support.
Don’t forget if you’ve enjoyed this podcast, why not of follow us through your favorite list listening platform. Also, please reach out to me directly at john@questionmark.com with any questions, comments. Or if you’d like to keep the conversation going, you can also visit the Questionmark website at www.questionmark.com to register for any of our many best practice webinars we host monthly.
And do check out Phillip Dawson’s books. He speaks very well and I’m sure you write very well as well. Thanks again everybody, and please tune in for another exciting podcast discussion we’ll be releasing shortly.