Welcome to this Questionmark podcast. Questionmark podcasts bring you news, ideas, and advice about assessments and learning.
Joan Phaup, Questionmark: This is Joan Phaup from Questionmark, and I’m speaking with Lynn Duffield, Senior Staff Specialist for the learning group at Verizon Business. Lynn, welcome.
Lynn Duffield, Verizon Business: Thank you, Joan. I’m pleased to be here.
Joan: Could you start off by telling me about Verizon Business and the work of the learning group?
Lynn: Verizon Business is a wholly owned subsidiary of Verizon Communications, and our end customers are medium, large, and multi-national corporations. We have 55,000 employees worldwide. The learning group is part of the HR organization at Verizon Business. We have about 135 people, and we provide the full gamut of training, from products, systems and tools, technical, HR, sales training, management training, etc., and our primary delivery mechanism at this point is interactive distance learning.
Joan: Can you talk a little bit about who the people are that you train and test and what their job roles are, and how many tests you’re giving?
Lynn: Certainly. Our primary audience are, of course, the salespeople. But we also provide training to pre-sales and post-sales employees, as well as our network operations folks. Just recently, we’ve been asked to extend our training activities to Verizon Wireless, Verizon (?1:35), and to our IT support folks. I’m not certain how many tests we currently administer, because we’ve recently changed the rules, and said that not every course has to have a formal assessment in Questionmark. But I do know that we have over 800 assessments in Questionmark, and these are level two, mostly, end of course assessments. There are some quizzes, and we’ve just recently started doing some level threes using Questionmark.
Joan: With all of this going on in your group, I know you’ve had to give a lot of thought to best practices in organizing test items. So could you explain why this is important, and what in your case are the key issues involved?
Lynn: Well, when we first got Questionmark software five years ago, it was, how fast can you get it implemented? And we spent more time planning how we would implement the software rather than planning how we would use it. So as a result, we’ve had to change the ways we’ve implemented our item organization. We started out with a topic-based and then moved to an organizational-based, but we found strengths and weaknesses in both. But the most important thing for each organization is to figure out how you’re going to use the software, and how you’re going to use it in the future. That’s one of the areas where we’ve certainly failed.
Joan: And how did you rectify that?
Lynn: Well, part of that is the change we made from a topic-based to an organization-based hierarchy. Our key issues are that we work in a very dynamic environment, that things change quite frequently. And we wanted to ensure that all of the objectives in a course were covered with the same amount of weight that they’re given during the course itself on our assessments.
Joan: You mentioned switching from basing your organization on topics to switching it to objectives. Could you walk me through the pros and cons of organizing items on the basis of topics?
Lynn: Right. The pros with organizing your items based on topics is that you have a much smaller database. What you’re doing with topics is you are reusing topics in multiple assessments. And one of the really big benefits of that is that your items are always current. As I said, we work in a very dynamic environment, and when something changes, the developer would go in, make a change to that one item, and then on every assessment where that item was used, we would have the most current version of the item. Some of the cons, though, are that our topics were much too broad. They were huge. For instance, in product training, we talk about the four P’s. We talk about the product features, we talk about the product pricing, product positioning against competitors and the processes to implement and sell the product. Well perhaps someone wants to do a course that just talks about the product features, and they would say, “Give me an assessment with product topic,” and we would have questions that related to product pricing or product positioning on that assessment. Also, we had a wide level of skill levels in each topic. For instance, we had beginner level course topic(?5:23) items, intermediate and advanced, and sometimes a very intro level course would have advanced questions on it. So that wasn’t a, that didn’t work real well for us. But our biggest issue was that we could not guarantee that all of the objectives in the course were covered, and covered to the same weight on the assessment.
Joan: Okay. So switching, deciding to make the switch to objective-based organization, what were the benefits you found of doing that?
Lynn: Well, basically you flip the pros and cons. We could guarantee that all of the objectives were covered, because the objective were listed as subtopics. And theoretically, more items would be written for the more important objectives. And when we create an assessment, we would pull randomly from each objective, and the objectives that had more items would have more items on the assessment. The big issue became with data currency, though. When something changed, the developer had to remember that yes, I have this sort of an item in, say, three different course topics or course objectives, and therefore I have to make sure I change all three. If they just changed the one, then the next time that the other two are presented, we have inconsistent items. Another couple of benefits are that you have much more detailed information for your participants. For instance, when you create your assessment, you can do topic feedback to the participants, and tell them that they scored maybe 86% on this one objective, and here’s the title of the objective. They scored 62% on another objective. So it is a mini coaching report at the time the assessment is given. It also provides management with more detailed information, because then management can take a look at the course as a whole, and see which objectives people aren’t doing as well in, and try and figure out why is that occurring? Is it because the material is incorrect? Is it, could be that the items are wrong, etc. But a big complaint that I hear from my developers is the topics are limited to 50 characters. So people who write really long objectives find that they have to be truncated.
Joan: [laughs] That’s a big problem. Are there other alternatives that organizations should consider as the basis for organizing items?
Lynn: One of the things that I wish we had investigated a little further was using meta-tags with topic-based. This would have allowed us to say that this is an easy question or a beginner level, intermediate, and advanced. The problem was when trainers would go out and deliver an assessment that had questions on them that hadn’t been covered in the course materials, the trainers would say oh by the way, the correct answer for this is A, B, C, or D, which messes up your item statistics and makes it difficult to say that this is a hard question or an easy question, because all of a sudden, everything was skewing towards easy. Subtopics are useful. You can use subtopics in topics, where you have your common items in your root topic, and then you can have subtopics below that. For instance, we could have done product, a product topic, and then contain the four P’s, product features, price, positioning, and process. The big thing is to talk to your developers and your trainers and get their input. Ask them what they think would work best. This also serves to educate them as to some of the issues you may experience, and maybe it would have eliminated some of those, the trainers giving the correct answers to questions that shouldn’t have been on the assessment.
Joan: What do you think are the key issues to consider in deciding how best to organize test items?
Lynn: The key issues as I see it are, you need to figure out what’s important for your organization. Do you want to ensure that all of the objectives are covered on an assessment? If so, you’re going to want to tend to go with an objective-based item hierarchy. Are you going to pull a subset of all the items? If so, you might want to stick with an objective-based hierarchy. Also, how much or how frequently will your items change? If you work in a very dynamic environment, you probably want to stick with a topic-based organization, just because of the data currency issues, where you change the item once and then every assessment that it’s used in will be accurate. Also, what kind of reporting do you want from your assessment and your items? The objective-based hierarchy provides you much more detailed information about the goodness of the assessment as a whole, how people are doing on the various objectives.
Joan: That’s great. That’s a lot to think about, and I really appreciate your sharing your experience with us. It’s been great talking to you.
Lynn: Well thank you very much for asking.
|
Podcasts 
