Welcome to this Questionmark podcast. Questionmark podcasts bring you news, ideas, and advice about assessments and learning.

Joan Phaup, Questionmark: Hello. This is Joan Phaup from Questionmark, and today I’m talking with Dr. Shannon Bonner, Corporate Testing Project Manager at Southern California Edison, about the goals and principles of good test maintenance. Shannon, welcome. It’s great to be talking with you.

Shannon Bonner, Southern California Edison: My pleasure.

Joan: Could you put us in the picture by telling us a little bit about how you are using assessments within your organization, and the value of good test maintenance to you as a corporate testing manager?

Shannon: I’d be happy to. Our company has a fairly robust testing program. We give about 15,000 pre-employment and promotional tests a year, so we’re very high stakes. These are tests that either grant people entry into the company in entry level positions or allow people to move from one position to another across the company. We have everything from, I think roughly 150 knowledge tests, which are asking content knowledge questions, to several dozen performance tests, where candidates have to demonstrate hands-on, specific skills, to hands-on software skills tests and even managerial assessments, and we’re using Questionmark to track if not to deliver all of those assessments, and to track results. It is our database of record.

Joan: What would you say are the goals of test maintenance?

Shannon: Well for us, the number one goal is that we’re going to maintain the underlying validity of the assessment. That is, we’re going to make sure that the assessment does whatever it’s intended to do, both for the obvious business reasons that I just mentioned and to make sure we’re reducing liability. So our goal is to make sure that we are keeping the test valid by keeping the test up to date and well maintained.

Joan: What are some of the consequences of poor or inadequate test maintenance?

Shannon: Other than those. Those are definitely things I’ve just touched on, so you know we’ve got the time and the money that’s wasted by forcing people to come in and test and then retest if a test wasn’t maintained. We’ve got legal challenges. We’ve got grievances when there’s tests that aren’t appropriately maintained and still obviously able to demonstrate validity. And then we’ve got the fact that with 150 knowledge tests, any time a candidate comes in and has a question or there’s a problem with one single test, it casts doubt across our entire test program. So, you know, that keeping every test up to date really helps the whole program look better.

Joan: What would you say are some of the key principles of test maintenance?

Shannon: A number of them, one being that test maintenance needs to be conducted on a regular basis, and obviously regular can vary from one test to another in that if you’re testing a knowledge test on content for a job that’s pretty stable, regular maintenance might be a longer interval than for a job that’s constantly changing. You know, we’ve got our meter technicians here. Meter technology advancing like crazy. We’ve got smart metering now, so that job is constantly changing. We have to do more frequent maintenance on that, than, say, our painters, and the field of painting has not changed terribly much in the last five or six years. So frequency would be one key principle. Another one for us would be that it’s documented, that it’s documented and trackable so we can show, here is what we did to maintain the test, here is how we did it. Here’s how we do it for all of our programs. So that would be the third, would be that we’re consistent, that we have a standard and that we follow that standard across all of our test maintenance for all of our tests. I think those are our key principles for test maintenance here at SCE and other organizations may have additional ones.

Joan: I wanted to ask you about exams that take place in diverse settings or with diverse candidate populations. Can you talk about what has to be done differently in those situations?
Shannon: In that case, the challenges are inclusion, you know, and diversity, I’m sure with some of the organizations that also use Questionmark are they have more diverse organizations than we do. You know, we can be talking about different departments or different unions. That’s diversity for us. But in other organizations, diversity may be even down to different languages. And then you’ve got to worry about inclusion. Are you representing all of the appropriate parties in your test maintenance process? So if you’re having to find content experts, are you getting your content experts to maintain the test to represent all of those different parties? You’re getting down to tests in different languages. And you’ve got the challenge of, if you want to maintain a test, it’s not as simple as making changes to an item. You’re going to have to make changes, translate it, go through whatever back-translation process you’re using. So it gets to be an extra layer of planning and preparation, and some extra work trying to include all of the stakeholders that you need for tests that are being used for diverse populations.

Joan: Thank you. How can currently available assessment technologies help organizations maintain their tests more effectively?

Shannon: I think we’ve got a lot of tools in our arsenal now for test maintenance, and it’s pretty exciting to see how easy it is to use some of them these days. Item analysis, test analysis reports, those are oftentimes a first line test maintenance tool or technology. Those let us see almost immediately, how are our tests performing? How are our items performing? Do we have problems? The ability to track candidate comments electronically and to see what comments we’re getting on items. You know, the candidates are often the ones taking the tests, are often the first ones to say, “Here, there’s a problem with this test. There’s a problem with this item.” So the better we are at tracking that, and the better tools we have to use to track that, the easier it gets. One of the things that I’m really excited about, that’s really useful, is the ability to use experimental items, which Questionmark has a product that provides, you know, is a great technology that lets us try items out. A candidate doesn’t know that it’s not a real item, so they put the effort into answering it correctly. And then we get to see, how does that item perform? That lets us build up a better item bank. Part of test maintenance for us is making sure that our tests are secure. If somebody finds out the answers or the questions, the test has now become invalid once the questions get out to the street. So question security is a huge deal, and it’s a huge part of keeping tests well maintained, and technology like Questionmark Secure, where we make sure that the candidates are only taking the test and not doing anything else, helps us with the security piece, and there in turn helps us with test maintenance.

Joan: Great. I’m going to finish off with a last question, and that is, what advice would you give someone who’s just beginning to design a test maintenance plan?

Shannon: Talk with others, would be my number one recommendation. You know, there are so many aspects involved, so I think, start talking with others. See how other organizations are performing test maintenance. What staff(?7:22) do they use? And start to make a checklist of what do you need to do in your organization? What would be appropriate for your tests? Look for best practices that you’re seeing across organizations, and then build from there.

Joan: Thank you. Good advice! Thanks very much for joining us today.

Shannon: My pleasure. Thank you.