 |
Welcome to this Questionmark podcast. Questionmark podcasts bring you news, ideas, and advice about assessments and learning.
Sarah Elkins, Questionmark: This is Sarah Elkins from Questionmark, and I’m very pleased to be talking today with John Dermo, who is the a-Assessment Advisor at the University of Bradford. In the last two years, John has been involved in two e-assessment projects at the university involving Questionmark Perception, and will also be presenting a session at the 2009 Questionmark European Users’ Conference. Welcome, John. Thanks for joining us today.
John Dermo, University of Bradford: Thank you.
Sarah: Now firstly, could you tell me briefly about the projects you have been working on at the University of Bradford?
John: Yeah, sure. Yes, and as you said there were two e-assessment projects we’ve been working on. First there was one in 2007-8. There’s actually something written about this in one of the Questionmark academic case studies. This was our Pathfinder project, funded by the Higher Education Funding Council for England. It was called Embedding Support Processes for e-Assessment. Basically, it was setting up all the necessary things that we would need to do to support e-Assessment once numbers increased in future. So training programs, setting up channels of communication, sorting out workflow models, especially for summative e-assessment. Also, we put together a policy document for e-assessment, some rules and regulations for computer assisted assessment that were approved at the highest level by university committees. This kind of paved the way for the project that I’ve been involved in more recently, which was funded by JISC. It’s called the ITS4SEA project, which stands for Integrating Thin Client Systems for Secure e-Assessment. So it followed on from the Pathfinder project, and it had a number of different deliverables and project outcomes, but the main one was that we built a new room, a new e-assessment room, with 100 seats, which was completed earlier this year. And this has really increased the potential for e-assessment at the university. All of this, both these projects and other initiatives have been made possible by the university’s e-strategy, which was 2004-9, and his has shown the institution’s commitment to e-learning. And they helped to set up two instances of Questionmark Perception, one for high stakes and one low stakes assessment.
Sarah: Now the e-assessment room that you’ve developed at the university, could you tell us a bit about the features that make this such a unique setup?
John: Yes. Well, as I said, it’s, you can get the hundred students in there, so it’s much bigger than anything we had before. But probably the main difference is that it uses what’s called thin client technology. So it’s not your standard PC. Students are using terminals, not PCs with hard disks. It’s all server-based. So students access virtual desktops. We did this because it makes the whole thing easier to administer technically. And it’s also, it’s easier to deliver assessments in a secure way. If we compare this to how we used to do this with PCs, where you have to install the software on each individual PC, this can all be done centrally now, which is much easier to deliver the desktop. And also it’s easier to only deliver what the students need for the assessment. So it might be a link to a Questionmark assessment, maybe a spreadsheet or some documents to read. But very easily, we can remove any of the links, so students aren’t tempted to visit other websites or use communication tools. So it makes the whole thing much more secure, more easily. Also, another aspect of the room that was quite innovative was the layout of the room. We wanted to move away from the idea of just standard rows of like parallel terminals, to create a more flexible learning space. So we worked with an architect from various designs. This was quite a challenge, (be)cause this really was quite innovative. I mean there is quite a lot of documentation about technical specifications for rooms, about minimum distances between desks and positions of monitors and so on. But there isn’t really so much out there about what’s best for teaching and learning, but at the same time making it secure for somebody’s assessment. So the room itself looks quite different, and people are often surprised when they see it. Another thing that is quite innovative, quite interesting, is that we’re experimenting with smart cards. So the students just have a monitor, a keyboard, and a mouse, but there’s also a slot on the monitor where they can put a card. And we can set it up so the assessment that’s delivered to them is identified by the card. So we could simultaneously have different exams going on in the same room simultaneously. So that’s quite interesting as well.
Sarah: And so the room provides a lot of additional security features. Do these help ease some of the concerns people might have about moving to e-assessment?
John: Definitely. As I said, the technical side, the technical setup is much easier, because it uses Virtual Desktop, so we can push out secure assessments remotely. It’s also much easier for invigilators, and they’re much happier with the setup for summative assessments. We use some software called Accents, which removes all the other icons and features of the desktop that aren’t needed. So this lets invigilators get on with the job of invigilating, rather than worrying about what students might be getting up to. Also, it sets instructors’ minds at rest, particularly in terms of security. We do use the randomization features of Questionmark Perception as well, to add extra security there, and the feedback has been very positive. I think administratively as well, it’s been quite a lot easier. We’re now looking at streamlining the admin process for summative e-assessments, integrating different systems like the student record system, exam scheduling software, and Questionmark Perception. So now, for example, we can send out personalized exam timetables to students by email, and we’re looking at automating at least parts of the process. Also, we’re now, by moving more to online e-assessment, we’re now able to look more closely at item banking in some areas, and using item analysis to ensure quality items, and also I think more efficient use of time. So yes, it certainly has addressed a lot of people’s concerns.
Sarah: So the room seems like a great setup for summative assessments. But what about formative and diagnostic testing?
John: Yes. Yeah, that’s a good question, because clearly, summative assessment only takes place during limited periods, normally two or three weeks towards the end of a semester, and we need to make sure that the room is really taken advantage of all the time. So we’ve also got to focus, and this partly informs the innovative layout of the room, focus on innovative teaching techniques, especially involving group work, formative feedback, and peer assessment. We’ve already used Questionmark Perception for this in the past, but we want to develop this much more. Of course personally, I feel this is where the real main impact of the assessment could be felt. And this is our main target for 2009-10, focusing on using this room for low stakes informative assessment, which links in with the university’s new learning, teaching, and assessment strategy for 2009-14, which has a big focus on formative feedback. I’d also like to get more involved in the research side of it, of formative e-assessment, to really measure the impact of the use of this room for e-assessment on student learning.
Sarah: Great. And so what has been the response from departments within the university?
John: Right. We first used the room in January ’09, towards the end of the project. We used it for two large-scale summative assessments. Then we used it in spring ’09 for some low stakes informative assessments, and then scaled it up to run it at least for one week at full capacity during our examination period in May. And the initial feedback from departments has been really very very positive indeed. In fact, particularly after the first two assessments, and they were both quite large modules that we deliberately chose for this, it attracted a lot of attention. We had lots of people contacting us, interested in using the room themselves. So the problem in fact we’ve found is that demand is really taking us towards the capacity of the room. So it’s a matter of sort of managing what we can do to try and meet the demand.
Sarah: And what about the students? How have they found using this room?
John: Right. We, towards the end of some of these lessons, we carried out sort of exit surveys with the students to gauge their attitudes towards it. And it’s been very very positive indeed. I mean, it’s clear that students do like doing online assessments at least as much as paper-based assessments, and in many cases they prefer it. And as for the room, we’ve had feedback saying that it is a very pleasant, it’s a very light learning space. I think the students appreciate this. We’re still working on various features based on some of the feedback we got. For example, we’re making it more adaptable by giving students foot rests and adjustable stands for the monitors and so on. But generally, it’s clear that there is an expectation among our students that technology should be used in teaching and learning, and I think e-learning in general as well as for assessment. And the use of technology in learning and assessment is seen as a good thing, certainly a forward-looking development, and I think that students do appreciate that the university is investing in innovative learning spaces in this way.
Sarah: Great. And so finally, have you had any challenges with the room?
John: Yes, certainly. I mean, I think because it is so new, so innovative, nobody has used this thin client technology in quite this way. It is used in other institutions, but what we’ve, but for different things. We’re the first institution really using it specifically for e-assessment. And one of the challenges we had was making sure that we had enough resources in the servers to deal with the simultaneous hit that you would get when the invigilator says, now start. It wasn’t so much the Questionmark servers, but the servers that were running the virtual desktops and the thin clients. And Sun, because we use Sun Ray technology with the thin clients, were very helpful in dealing with some of these. So it’s a matter of sort of weighing up the resources that, it was very difficult to predict exactly what was needed. Also, I think another challenge, as well as the technical challenges, was trying to balance the very different needs of formative and summative e-assessment within one room, within one facility, because we’re finding already that during the May examination period, we were already running the room for summative purposes at about 45% capacity. So, you know, it only requires a sort of doubling it and a bit more, and we’ll find that we’re already reaching capacity with that room. So I think we need to focus on using summative e-assessment where it really adds value, so on very large modules or where assessments are being used which couldn’t easily be done on paper, and also focusing more generally on the room, using it in a more flexible way for formative e-assessment throughout the academic year.
Sarah: Well thanks, John. It’s been great talking to you today, and I hope others can learn from the work you’ve been doing at the university.
John: Thank you.
|
Podcasts 
