Welcome to this questionmark podcast. Questionmark podcasts bring you new, ideas, and advice about assessments and learning.
Sarah Elkins, Questionmark: This is Sarah Elkins from questionmark, and today I’m talking with Greg Pope, Questionmark’s Analytics and Psychometrics Manager, and Stefanie Moerbeek, who is the Senior Coordinator for Examination Development at EXIN. Stefanie and Greg recently facilitated a peer discussion on hot topics in assessment at the Questionmark European Users’ Conference, and they are joining me today to share the results and findings that came out of that session. Hi Stefanie, hi Greg.
Greg Pope, Questionmark: Hi.
Stefanie Moerbeek, EXIN: Hello.
Sarah: Okay, so can you start by telling us a little about the session and what you hope to achieve?
Stefanie: Well, we wanted to facilitate a discussion among our peers, and on several topics that are of great interest in the assessment community, and we wanted to bring together theory and practice, and also bring together different types of users and experience. We wanted to discuss and share and learn from each other, and broaden views, and learn about practical choices made in different areas of test use, for instance, academic and corporate and (?1:14)ing bodies.
Sarah: So can you give us a brief overview of the topics you presented, and why they are important in today’s testing and assessment environment?
Stefanie: Well, we had four topics on our agenda, but there was a lot of discussion. We were only were able to discuss three in the end. We didn’t get to discuss the psychometric rules for building defensible questions and assessments, but we did get to discuss effective techniques for beta testing of questions, and using random administration of assessments, and dealing with intellectual property theft of exams.
Sarah: And the beta testing of questions generated a lot of discussion about fairness issues, and the practicalities of completing such tests. Can you tell us what sort of issues were raised, and the suggestions for overcoming them?
Greg: Sure. So there was a great deal of discussion about whether getting participants to take beta questions that don’t count towards their assessment score is appropriate, and that was hotly debated. It was felt that participants definitely needed to be informed if they’re taking questions that aren’t going to be scored towards their final marks, and in general it was felt that it would be better to intersperse beta questions within assessment, rather than having a separate assessment, but that the participants should not know which beta test questions are the ones that are going to be unscored. There were also suggestions that beta tests might be best conducted in formative context, but questions were raised about the generalizability of participant performance in this context. People generally felt that beta testing had many benefits, such as obtaining psychometric performance information regarding questions to help build more defensible assessments. However, there were some practical issues that were debated which make beta testing difficult, such as the expense of developing more questions. Adding more questions to an assessment takes time, adds time to the assessment, and of course the (?3:11) and fairness issues that I discussed.
Sarah: Many people like to use randomization in tests, but this can have an impact on the robustness of the actual tests. What type of issues were brought up during this session?
Greg: So the issues mainly centered around the defensibility of randomly administered assessments, and the challenges with psychometric reporting. So there is a discussion about whether a random administration of questions from topics would result in equivalent test forms for participants, for example. Many attendees felt that more parameters should be required to pull questions through(?3:46) topics, such as choosing the number of percentage of questions within a topic by various attributes or metatags. Computer adapted testing was also discussed as a possible solution. The challenges regarding psychometric reporting centered around not having enough participants answering questions, as every participant would not be answering every fixed set of questions. Rather, you might have 200 participants. Rather than having 200 participants take all 50 questions, you might have 10 participants taking 150 questions. And this is something that you unfortunately can’t really solve with better reporting.
Stefanie: I’ve seen(?4:23) we use sort of a mixed form. We use the functionalities of randomization in Questionmark to create fixed sets. We recently found for creating some new exams that we would like more criteria and more metatags and combinations of topics and percentage of items from topics to create the test, and we weren’t able to. So we would like to see more functionalities like that.
Sarah: Stefanie, you shared an example of a brain dump of some of EXIN’s exams. Can you tell us what happened there?
Stefanie: Well, I found a couple of months ago, a forum, a Dutch forum on the internet, where our candidates, or some of our candidates, discuss our exams. And one of the exams, I found not only discussion on what books should I study; what materials should I study; what kind of items can I expect? Does anyone have any sample material? But I found part of an exam, as a brain dump. It was about two thirds of an exam about (?5:40)ation systems foundation, and not only were the questions there, but also discussion on what the right answers should be. So basically, one of our exams is gone. It’s out there. And it’s not, this is just one example. I found another one some time ago, and there must be more out there.
Sarah: Now Greg, you do a lot of work with the Association of Test Publishers Security Initiative. Can you give us a quick overview of the recommendations that have been defined for dealing with IP theft and other security issues?
Greg: Sure. So some of my colleagues on the ATP Security Initiative, namely Joe Canada and Kathy Donnett, put together a fabulous document outlining best practices in preventing, detecting, and reacting to the theft of exam content. This document discusses techniques of using webcrawl programs, writing cease and desist letters to website administrators, and writing candidate agreements. I also do some work with the Test Security Initiative on developing a test security plan for organizations that are administering high-stakes assessments and have strong security requirements. So these documents are available to ATP members and are great resources.
Sarah: And what do the delegates have to say about this topic?
Greg: Generally, people in our session felt that removing the market for stolen intellectual property was a good solution to this problem. If practice tests and example questions are made available to participants before they take an actual high-stakes exam, it was felt that there would not be so much pressure or demand to steal the intellectual property off of exams. And Questionmark now allows for the selling of assessments, recently announced as the technologies for selling tests. So an organization can actually sell practice assessments to bring in revenue as well as combat the theft of intellectual property.
Sarah: Great. Thanks Greg and Stefanie. It’s been great talking to you today, and you’ve given us a really good overview of the current issues facing the assessment industry.
Greg: Thanks.
Stefanie: You’re welcome.
|
Podcasts 
