Welcome to this Questionmark podcast. Questionmark podcasts bring you news, ideas, and advice about assessments and learning.

Sarah Elkins, Questionmark: This is Sarah Elkins from Questionmark, and today I’m talking to David Lewis, who is the Blended Learning Coordinator at Glamorgan University. Hi David. How are you today?

David Lewis, Glamorgan University: I’m okay, thank you Sarah.

Sarah: Now can you tell us a little about your role at Glamorgan University?

David: Yeah. I mean the Blended Learning Coordinator, really, it’s a title that could mean almost anything. But what I have to do at Glamorgan is, I act as a kind of liaison between our academic policy unit, between our central computing and library facilities, and between the academic faculties. So I spend a lot of my time kind of selling out our services as a support department out to those academic faculties.

Sarah: Now you’ve said that assessment is at the heart of learning. Can you tell us why you see assessment as such a crucial part of the learning experience?

David: In my own particular institution, we’ve got a very strong history in providing things like online learning and a lot of very innovative teaching methods mediated through electronic means. However, the thing that really was missing when I came was any focus on online assessment or bringing assessment forward in the way that it really needs to be brought as an essential part of any sort of online learning experience. And this is something that we’ve really kind of changed. There’s been a real sea change, shall we say, in the last three years, in my institution in terms of our institutional strategy. So because we are funded by the Higher Education Funding Council for Wales, they expect us to put periodic strategies together to look at our teaching and learning. And the latest strategy is actually called the teaching and learning and assessment strategy. So for the first time really, assessment is front and center with regard to our policy and strategy in the universit.

Sarah: And now over the last three years, you’ve developed a large-scale implementation of online assessment at Glamorgan University. Could you give us a brief overview of the way Glamorgan is now using online assessments?

David: Yeah, sure, of course. When I joined Glamorgan about three years ago, there was a couple of pilot projects looking at large scale and phased implementations of online assessment within particular modules. When I joined obviously with like a dozen years experience of online assessment, I took control of those projects, and we worked very closely with the academic members of staff to develop the way that they were actually going to deliver those assessments. So for instance, our probably most successful pilot was one where we looked at the accountancy students in their third(?2:47) year, and how they went to taking phased tests every month during their first year. What we did was, we delivered those tests online, so it meant that the students were able to get their feedback immediately at the end of those phases, so that they had plenty of time to prepare for the next phased tests. You would imagine if somebody was marking those manually, if there’s only a four week gap between each test, it could be two, three, or even four weeks before somebody gets their feedback. Well in terms of developing their learning, that really is a barrier. Questionmark enabled us to really remove that barrier completely, and since then, the students have got a very positive experience of online assessment. Some research I’ve done recently with those students shows they’ve got a substantial appreciation rate in for that form of assessment, and I would suggest it’s because they’ve had this very positive experience. The other project that we did in the pilot was where we actually looked at the final year module that had been previously assessed wholly by essay. The member of staff concerned wanted to replace that completely with an online assessment. Now we kind of negotiated that down so that the online assessment was about 75% of that module mark. But still, for our institution that was a fairly big step. To completely replace a final year module assessment with an online assessment was, for Glamorgan, a real leap forward in terms of our assessment strategy and policy. And having approximately(?4:16) 120 students all taking those assessments at the same time, that was the first time. That run was a very scary time for us, but, you know, with a very positive outcome. That was about two years ago. Since then, we’ve developed our systems. So initially we had a single server. We realized because people use the assessments in different ways, we now have two separate servers, one that we call our formative server, which is very closely integrated to our VLE Blackboard, the other which we call our summative(?4:48) server, which has got a slightly higher level of security, and is less closely integrated with our Blackboard server. The purpose for this is it means that for instance, on the formative server somebody can deliver instant feedback to the student directly linked in from their Blackboard modules. On the summative server, the feedback is under the control of the module leader. So if the module leader only wants to send out the scores of the students, they can do that. If they want to send out a full feedback report by email, they can also do that. So they’re slightly different methodologies underlying both those servers. And that really works for us. Since then, we’ve realized that we can put Questionmark to a lot more uses than simply the kind of classroom or exam room type of assessments. So we do a lot of survey type work. We’re not talking about, you know, traditional sort of, (?5:40) type surveys. We’re talking about surveys which are also providing the student with some valuable experience and valuable feedback. I’ll give you a for instance. We do a thing that we call the early days questionnaire. And the purpose of that is to talk to the students about their experiences of being a Glamorgan student within their first two months’ experience of the university. And what happens is, the students get us some general questions. So they may get those kind of demographic questions like, you know like, you know, are you male, are you female. Are you a UK student or an overseas student? But then, they might get questions like, “Do you know your faculty?” Which is fine. You would imagine that everybody knows which faculty they belong to. But sadly for us, we do get the occasional student who slips through the net. If the student answers no to that, and they say they don’t know what faculty they’re in, they actually get some feedback which says, well, okay, this is a very quick and easy way of finding out what your faculty is. But you also should actually make contact with your faculty. You need to engage with your faculty. So in that way we get information about the students, and it helps us to mediate those issues. But from the students’ point of view, they get very valuable feedback and it really lends itself. And we read thousands and thousands of responses to those questionnaires, and it’s been one of our most successful projects.

Sarah: Could you explain the difference between what you use for formative assessment, summative assessment, and module evaluation, and why you chose to do this?

David: Yes. We’ve also got a server that we use wholly for module evaluation. I went to a Questionmark presentation from Tim Ellis from Lancaster, where they talked about how they were using Questionmark for module evaluations. Now we really took that to heart in Glamorgan, and we have implemented a system that allows us to do that as well. And that again is based on a Questionmark server. It’s really, really useful, because it enables us to very quickly write the questions to present a single, central, agreed form of module evaluation for all of the students. Additionally for us, we have a Welsh version of that. So we’ve actually got a very interesting feature for us, which is, you can get the sort of introductory information about the module evaluations. And they get a link which says, “If you’d like to have this is Welsh, please click this button.” And immediately, the language translation changes. Very, very useful for us, because as an institution, we have to engage with the Welsh language. And then all of that information is held centrally by obviously the Questionmark server, but then is fed back to the module leaders through Blackboard. So literally, all the module leader has to do is click on a very simple link within their Blackboard module, and they get a survey report delivered back to them which is all of the responses that the students have made. That’s been extremely popular in Glamorgan, and you know it’s been a real useful feature for us because it’s a, you know, it takes the burden of that kind of evaluation away from the staff. We need, obviously, to make sure that security is preserved, and that the module leaders feel there’s no kind of big brother or you know that there’s any hidden agenda in this. It’s about actually providing a really useful facility for them that gives them the ability to do this, you know, at any time, with students who are on campus, who are off campus, you know all the same issues that lend themselves to assessment online lend themselves to module evaluation online. And again, that would be a thing that, we’ve built on our experience. We’ve built on our servers. We’re onto our third server now. Who knows where we’ll end with this. The difference is, I’ll come back to your question, between the summative and the formative, effectively the licenses on those two servers are identical. We’ve gotten unlimited licenses on both those servers. But it’s the way that they actually link or the way that they interact without our Blackboard server that’s the difference. The formative server has got the Questionmark Blackboard building block bridge on it. So it means that for the module leaders, it makes it very easy for them to deliver their assessments, because it’s only a few clicks, and the assessment is there, delivered for them within Blackboard. And similarly for the students, they only have to click on the link, and the assessment is delivered to them. Once that assessment has been completed, they can click on the link in their Blackboard gradebook and get their feedback. So that integration, that very close integration is what we consider to be formative assessment. The instance of feedback that is provided to the student, the ease of integration between those two systems, and that’s our formative server. The summative one has got a little bit less integration, and it’s quite deliberate that we do that. So for instance, the students do not get to go back and look at their feedback reports, their (?10:31) reports on that server. That functionality is deliberately switched off. The reason for that, and I mentioned this earlier on, is we have phased tests. So very often, a module leader may deliver the same assessment on a Monday, on a Tuesday, on a Wednesday of the same week. The problem being, of course, is if people on the Monday were able to share their feedback with people on the Tuesday, they disadvantage the people on the Tuesday or the Wednesday who hadn’t seen that. So what we encourage the module leaders to do is, is once they finish those phases, they can use the email functionality to email out the (?11:02) reports for those students. And they get those (?11:04) reports delivered into their email box. So there’s no difference in their experience. We’re giving an equal amount of feedback to those students that they need. But it’s not disadvantaging the students who actually haven’t had that assessment yet. And the way that’s integrated into Blackboard is very much on a kind of lighter, less structured integration. So effectively all we do is we provide a URL from Questionmark to Blackboard. When the students click on that they have to log in again, and it means then that there’s an extra level of security. What we tend to do is encourage the module leaders to make those window pop up out of blackboard, so that when they look at the student’s desktop, they can see that everything else is shut down on that desktop apart from their Questionmark test. So that gives us a kind of extra level of security, an extra level of confidence. And when people are going round those classrooms doing their physical invigilation(?11:58), it means they’re going to very easily see on the screen what the students are doing.

Sarah: And you’ve developed extensive training programs and support networks at the university for people involved with online assessment. Why are they so important?

David: Yeah, I mean that was part of our project aim from the very beginning, that we wouldn’t just have this product kind of sit in there, and we’d do it all for the student, for the students or for the academics really. What we wanted to do was make sure that there was a support network so that people had issues or they needed to engage with us, they could do that. But equally, if we could provide them with those skills, and they could almost go off and do it themselves, that we’d also be happy with that. And that, I think, has been the greatest success from my perspective, in that there are people who have really taken this and run with it. The training course that we do is very much based on the pedagogy of creating good questions and good assessments. It’s a half day course, so we take people through the way that they can create questions in Questionmark, assessments in Questionmark, and the whole pedagogy of creating online assessments, which is obviously going to be different from the way they’ve done it before. We also look at some of the research and some of the kind of publications to do with writing good assessments. So I mean, (?13:21)’s publications, for instance, who’s an ex Glamorgan member of staff, we refer to those in actually encouraging people to think about the way they’re delivering those assessments. In our second training course, we show people how they can actually deliver those assessments through Blackboard in our case, and some of the issues they might have in terms of how they might deliver those assessments. So we touch a little bit on classroom practice, on feedback. And we also cover reporting in that, so the way that the academic staff can see what the students would see, so we get them to try an assessment as a student, and look at their reporting as a student, and we also then get them to have that all all in view, so they can look at the whole class of results in a (?14:01) report, at an individual student’s results, at a question analysis report, so they can look at how their questions are performing. We do that whole range of reporting and analysis with them. And those are the two main training courses that we do. But we also do ad hoc training. We very much do awareness rating, so most of our faculties are into staff development. And every year, they’ll take their staff away for a day. That might be the whole faculty, might be an individual division. And we get involved in that kind of thing where we’ll go along and we’ll give a presentation, answer some questions, and very much sell our services, or sell the principle of online assessment to those staff. We also, in our strategy and policy unit, have got members of staff there who look at assesment across the institution, and they do a very similar thing. They sell out the principle of providing that assessment online. And in their case, it’s not just Questionmark that they look at. It’s the whole range of assessment types that people might engage with. So things, simple things like submitting their essays into Blackboard, using things like, turning it in to look at the potential for plagiarism checking, or online marking. So it’s about sort of engaging across the faculties with as many people as possible, in as many diverse ways as possible.

Sarah: Okay. And you’ve also developed processes for ensuring robust and reliable assessments in labs. Can you tell us a little bit about these?

David: Yes. I mean, this has been one of our major concerns from the very beginning, that we don’t die by our reputation, you know, that you’re only, with anything that you do, you’re only ever as good as your last failure. And we don’t want that to be something that people talk about. So we’ve tried wherever possible to ensure that everything that we do is as robust and reliable as possible. So that, for instance, I mentioned to you in the very early days, we did that with 120 people kind of sitting their assessment at the same time. We’ve gone away, maybe, from large groups in large rooms, and we’re looking now more at kind of 30-seat rooms, because that is the normal default computer lab size for our institution. And what we would try and do is, we would try and encourage people to have a kind of staggered start, so to have, you know, three or four rooms booked for their 100, 120 students, and start off the first group, and then move onto the second, and then move on to the third and move on to the fourth. We find that much better, because trying to manage all of those students, trying to manage 90, 100 students in one go is actually quite difficult. And managing the 30 students is much, much easier. So that’s not a technological thing. It’s more of a logistical thing for us in terms of what we do and how we do it. And I think it gives the academics more confidence, because they can sort of start one room off, be confident that that room is running, and then move on to something else. What we tend to do is when we do those kind of large-scale assessments, is we’ll have a member of staff from the central support team. Very often it might be myself. We’ll also have the academic member of staff there. We also may have a member administration staff from the faculty. And in our case, many of our faculties have really engaged with that. They’ve looked across the whole faculty, how their support services within the faculties can engage with this online assessment agenda. And then commonly, we’ll also have invigilators. So these are paid members of the public who come in and supervise assessments for us. So what we would do is, we’d start off the group. We’d leave the invigilators in there. We’d move on. We’d kind of float around. There’s, you know, 100 people, 120 people, whatever it is. And that’s the best way we’ve found of actually delivering it. It’s not the only way, I must add, but it currently, if we were recommending something we’d do that. And we’ve never really had a failure that we are, we were disappointed with. We’ve had issues in terms of trying to manage too many people at the same time, our infrastructure and things like that. But generally, when we do our assessments, they all come up very positively. I’ll give you a for instance. Recently, one of our most adventurous members of staff is a lecturer in computer science, and he’s doing exams (they might actually be going on at the moment, I don’t know) where he’s taken sort of 150 first-year programming students, and they’re all doing their exams online like that, which is fantastic. But he sort of came and grabbed me from my room, because fortunately or unfortunately, you might say, I live in an office that’s only about two doors away from him. So I’m very convenient for him, I am, really. Anyway, he came and saw me. He said, “Look, I’ve got this class, and we’ve got like 50 students, and we’ve got two students here who can’t actually submit their assignments.” So we had a look, and we got one of our people up, and it turned out that it was actually a dodgy wall socket. You know, the half(?18:46) cable and the wall socket weren’t actually working properly. So all the person did was, they unplugged the cable from the wall, plugged it back in, resubmitted their assessment, and it all submitted perfectly. So there is, there’s a robustness there that we don’t realize. (?19:01) underlying kind of technology that even we try and ignore, and really has worked for us in that way.

Sarah: So do you think this gives you any advantage over other higher education institutions in Wales?

David: I think, what I would like to kind of maybe claim, I don’t know whether our peers would do this, but I think they perhaps grudgingly would, is I would say that Glamorgan is really taking a lead in Wales in this type of assessment. But that doesn’t mean to the detriment of all our peers in Wales, because we’re very collaborative institutionally, between the different partners. And we’ve helped people. We had feedback from Aberystwyth last week to thank us, because they’d just done their first large-scale exam, and we’d helped them. We’d given them some time, some advice and consultancy on the way that they could do it. And I know Cardiff have just appointed somebody. And again, they were coming to talk to us about how they might structure their systems, how they might structure their training, the kind of how we’ve achieved what they would like to achieve. So I would, I’d like to claim that we’re a kind of class leader within our principality in that way. But that’s not to the detriment of other institutions. We like to help and collaborate with these people, because we feel that in the future, people might help us. We might help each other. It may even lead to some kind of national initiative. So I think in that way we’re quite pleased to be a leader.

Sarah: That’s great. And so if there was one piece of advice that you could give to someone starting out with online assessments, what would that be?

David: I think the most important thing that people have to do, is they have to have a mock version of whatever assessment they’re doing. So if it’s an in-class test, it could be a fairly simple, you know, formative, untimed, free-for-all kind of assessment. If it’s an exam, they should have a mock exam running before that. I think that the greatest fear and the greatest risk for anyone is you go in cold into any assessment, never having done that thing before, and that would be your greatest risk. So I would say, you know, try it out first. We very much encourage people when they begin to start off by writing some perhaps simple, formative assessments, to increase the level of complexity by doing in-class tests, with maybe the ultimate aim to doing large-scale exams. But going straight from nothing to exams would be not something that we would encourage and, you know, we don’t, and we’ve found that the methodology we’ve suggested is actually the most successful one for us. So try it out first.

Sarah: That’s great advice. Thanks David. It’s been great talking to you today.

David: Thank you.