Welcome to this Questionmark podcast. Questionmark podcasts bring you news, ideas, and advice about assessments and learning.
Sarah: This is Sara Elkins from Questionmark, and today I’m talking with Tim Ellis, who is the E-learning Development Officer at Lancaster University. Hi Tim, how are you today?
Tim: I’m fine, thank you. I’m happy to have this opportunity to come and talk about what we do at Lancaster.
Sarah: Great. Now before we start talking about your use of Questionmark Perception, I’m interesting in finding about Lancaster University and your role there. Well, Lancaster’s a medium-sized research-led university. We’ve got approximately 12,000 students. Our academic departments are grouped together into five faculties, so there is the Arts and Social Sciences, Science and Technology, Management, Lifelong Learning, and we’ve also just started a new school of Health and Medicine as well this year. We offer quite a diverse range of subjects for degree at both undergraduate and post-graduate level. My role as an E-learning Development Officer is that I work in a small learning technology group which is based in a central unit called the Center for the Enhancement of Learning and Teaching. Our job is to develop and support the university’s virtual learning environment and all its associated tools. As our VLE, we use a bespoke system which is based on Lotus Notes/Domino, which means that because we control much of the development of that, we can be much more responsive to the needs of the university and the departments. We pretty much support the whole process from server side, server hardware setup and its maintenance all the way through to software development and the staff training and consultation side of things.
Sarah: Okay. And you recently received the team at the ALT conference. Can you tell us about that and why you received it?
Tim: Yes we did. It was fantastic they recognized the hard work and quality of the work that we do here at Lancaster in the Learning Technology Group. The panel felt that for a small team of just seven and a half people we were able to deliver a complete e-learning solution to the whole university, comprising of pretty good quality products, responsive support, and developments which made a significant impact on the teaching and learning at Lancaster. So it’s always good to receive recognition from your peers within the field.
Sarah: Yes, fantastic. Congratulations.
Tim: Thank you.
Sarah: And so can you tell us about why you decided to use Questionmark Perception?
Tim: Of course, yeah. It began probably back in 2002 before I started at the university. It started as a single department, really, who had a requirement to deliver formatives assessments online for their students on a regular basis. They went away and found that Questionmark was the tool that gave them the power and flexibility to do what they wanted to do. So after they successfully purchased Questionmark and had it running for a couple of years, more and more departments around the university started showing interest in what was going on, and it was decided it was probably a good idea to move the support of Questionmark Perception into a more central unit like the Technology Group, where we could support and develop it in a much more robust way. That’s when I started, in about 2004, when we began improving the hardware which supported Perception and went about promoting online assessments to the university in general, really, and every year we get more and more people using it in this way. We’ve got quite a few departments now who use it in a variety of ways, mainly for formative or low-stakes (?3:41) assessments, but also for their module evaluation system that we now use.
Sarah: Okay. And over the last few years, you have been working on the Lancaster University module evaluation system. Can you tell us a little bit about this project?
Tim: Yeah. Well it came out of a review that was done by the main linear teaching committee at the university that showed that the module evaluation across the university was fairly ad hoc, and the student feedback on modules was performed in quite a number of different ways across campus, with a diverse range of questions and quite significant time costs on departments or administrators, whose task was often to print out the evaluation forms to then deliver them to the lecture or seminar, collate all the results together, which often means hand-typing up the free-type(?4:26) responses that the students gave. And for a course that was upward of 300 students, this is quite a significant time cost to them. So the review concluded that essentially administered systems that could deliver questionnaires online that could reduce the workload on administrative staff would be very beneficial, and would mean that the whole process was more systematic and consistent, particularly from a student point of view. At the Learning Technology Group, we’d already successfully piloted a system of questionnaire delivery using Perception with a couple of departments. So this proof of principle basically meant that it was a quite small step up then to build a system using Perception as the base for delivering the questionnaires. The requirements for the system were that they had to be as automated as possible, but we also needed to give the departments the flexibility to make the modifications that they required, and also would also need to integrate closely with our central student record database to get the information about modules and about students who had registered on those modules. Questionmark itself couldn’t do everything that we wanted out of the box, but because Perception is so flexible and configurable, we were able to write software which interacted with Questionmark very closely, delivered the questionnaires and collate the results. But we also used the same system to integrate with our own central databases. So we started the project in 2005, 2006. We piloted it across campus for the academic year 2006-7, and then after we had some consultation with the teaching committees and departments, made some quite significant modifications, and did a second pilot in ’07-’08, after which the university then decided to make the module evaluation system a policy that all top modules are now evaluated in this way, both undergraduate and post-graduate. So as it stands now, the system automatically picks up all the modules taught at Lancaster. When they finish, the students are then sent an email which contains a link to the questionnaire that they need to complete. Each questionnaire contains a core set of questions which are asked of all modules, and then there’s a second set of questions which departments could customize to their own requirements if they wish to. Each response by the student is anonymous, and can’t be tracked back to that particular student. But we do track which students have and haven’t responded to the evaluations so that we can actually send reminders to those students at regular intervals. Generally, the evaluation process runs for about two weeks, after which we sort of close it down. There are no more responses that can be made to that particular module. And then the results are collated into a report which is automatically emailed to the course leader and the head of department. So that means basically from start to finish there’s no requirement for any manual intervention, unless of course departments want to modify the process in any way. There are various interfaces where departmental staff can customize the system, including modifying the questions they want to ask, changing the dates when the emails are sent out, and also customizing the report and who actually receives the report from the system.
Sarah: Great. So can you tell us a little more about the integration with student records?
Tim: Yeah. As many systems at Lancaster, the student record system is a bespoke system, so it’s one that our system team have written themselves, with a back-end SQL database. So with close collaboration with our system services, we’re able to set up some scripts that could automatically synchronize the data that was in the student record system with our other e-learning tools, including the module evaluation system, but also online assessments, the VLE. It has been quite a major factor in ensuring that the data in the central records is actually accurate. So we pick up which students are registered in a module. We can tell when a module ends, and also which teaching staff are associated with different modules. So it is very well integrated into all our tools.
Sarah: And how have the students responded to the system?
Tim: I think they’ve responded quite well. We don’t get to see the reports, of course, from the module evaluations themselves, so we can only judge the students’ opinions by the response rates that we get, and by talking to departments. We see in the last couple of years the response rates for the module evaluations as a whole have been increasing year on year, so, which we hope is down to the students seeing that the ease and benefit of responding in this way is better. Talking with departments, there’s also, they are also seeing the benefit from the reduce workload from the administrators. So they can see that fast access to results, and the whole process is much more streamlined and easier to interact with. Although in many cases the actual response rates on the system are lower compared to the paper-based systems they were using previously, speaking to staff, the actual quality of responses they’re getting back is much better, particularly the free text responses, where the students tend to be more considered in their responses when they’re typing things on the computer rather than to rush and write them down on a paper-based system.
Sarah: And so were there any issues you’ve encountered or lessons learned that you’d like to share?
Tim: Yeah. One of the biggest issues we encountered was actually the quality of data that was in our record system. It was often the case that modules were recording as ending at the wrong time. It may have been when they rented the system. After that time the module had changed, had been changed, had been moved around, but not actually updated in the system. So this meant that in a number of instances, module evaluations were sent out at the wrong time. So this is the main message that we try to get over to departments now is that in any system like this, it’s entirely dependent on the quality of data that’s in the system. One of the things we implemented to counter this was that we now send an alert to the module leader a week prior to the start of evaluation, which explains the process, and explains when the time to get the module evaluation is. So then it gives them a chance to actually respond to them and say no, this is wrong, or, yeah, so we actually have something in place in between, sort of like the module evaluations, so we can pick out anything that’s not right in the data. The other major thing is that although a system like this is administered centrally, it’s important that departments actually engage with it and feel that they’re actually part of it, and feel ownership. If the departmental staff don’t engage, then the students in the department generally are less likely to respond, and we’ve seen this in, if you look across the departments, you’ll see this in the general response rates for those departments, that if a department actually engages in this, and engages with the students and actually talks to the students about the module evaluation and how they place high priority on the module evaluations, then you tend to find the students will respond much better. This is what I mean, is why we built such a high degree of flexibility into the system, so that departments could actually modify and feel like they retain ownership of the whole thing. So those are the main two points really that I think we’ve learned from this whole process.
Sarah: Okay, and now that you’ve all this experience with Questionmark Perception, what plans do you have in regards to online assessments at Lancaster University?
Tim: Well, there are still a number of developments to the module evaluation system that I’ll intend to implement over the next year or two, mainly just trying to integrate the system into the VLE more, and the same applies to online assessments too. Departments are wanting to deliver quizzes and tests from within the VLE pages themselves. So I’m going to be looking into ways of being able to do this, along with promoting online assessment in general across campus to try and increase the user base even further. If I can get online formative assessments embedded into more and more modules, than that will be great.
Sarah: Fantastic. So thank you Tim. Thanks for being with us and sharing your experiences. It’s been great.
Tim: Okay. Thank you. |
