Joan Phaup, Questionmark: This is Joan Phaup from Questionmark, and I’m talking with Karen Chisholm and Keith Peterson from the Arkansas Department of Career Education’s Office of Assessment and Curriculum. It’s great to have you on the call today.

Keith Peterson, Arkansas Department of Career Education: It’s great to be here, Joan.

Karen Chisholm, Arkansas Department of Career Education: Thank you, Joan.

Joan: Now, I’m going to be talking with you about how you process data, but before getting into that, I’d like to ask Karen a few questions. Could you tell me about the Department of Career Education?

Karen: Sure, Joan. We are a state agency that administers career and technical education in the state of Arkansas.

Joan: And the role of your office there?

Karen: We coordinate the development of all curriculum and assessment for the secondary level career and technical education in the state of Arkansas.

Joan: Great. And could you tell me a little bit about how you use Questionmark Perception for that?

Karen: Sure. We assess career and technical education students at the secondary level at the completion of a course, so it’s an end of course type of assessment that we’ve developed. We use our statewide results from those assessments to, as a reporting device to our federal agencies for federal regulations. We assessed, in 2008-2009 school year, we assessed about 104,000 students.

Joan: That’s a lot of people to test. How do you manage registration and scheduling for that many tests?

Keith: Joan, we have a website for our office that all of our public school teachers and public school administrators can access to learn information about our assessment program, our assessment schedule, and so on and so forth. Through that website, we’re integrated with a custom interface, designed for us by our Questionmark consultants. Teachers can access that custom interface, log in, choose the course assessment that they wish to take, and when they do that, they create a class roster. So in Perception lingo, essentially what they’re doing is creating a group for themselves to which they will register participants, which would be their students. So each teacher is responsible for going into our custom piece, logging in, and selecting which assessment they want, and then entering those students that would apply directly to their class that they want to have assessed. We have an automatic scheduling component that’s a part of that custom interface, that once they enter the student, it sets them up with the correct assessment, date, and time, and provides them with a password. And then we also have a login on our website through which the students can go in to access those assessments.

Joan: Could you talk about the assessment results and their importance with regard not only to students and teachers but also to your department as a whole?

Keith: Sure. Obviously, to the students and teachers, the main importance of our test result is going to be classroom improvement. We have, through the use of Questionmark reports as well as several customized reports that we’ve developed, we’re able to provide pretty immediate feedback to teachers regarding how their students performed in relation to our curriculum that we provide for them. So just for lesson planning, classroom improvement, and for students’ actual career and technical skill attainment, the tests are vitally important. But to us as a department and an agency, really probably the primary importance of the data that we get is our use of that data in providing it to the federal government. Career and technical education around the country in every state is funded by a Carl Perkins grant, which comes to us from the federal government. The state of Arkansas uses assessment data as one of its Perkins indicators to demonstrate to the federal government that we’re not only teaching certain coursework and curriculum, but we’re actually making efforts to improve student skill attainment in relation to that coursework. So for us, it’s not only important that our students and teachers continue to do well, but that data is important in us receiving literally millions of dollars a year in order to fund our state programs.

Joan: So in order to do all this, you must be processing a tremendous amount of data. Could you offer some pointers about how to handle that?

Keith: Sure. You know the, when you do handle over 100,000 students who take 60 60-question assessments in about 50 different courses throughout the course of the year, you get a lot of data that, I won’t say unnecessary, all data is useful, it just depends on what your use for that data is. So in order to handle it efficiently, you really need to come up with some clear objectives ahead of time as far as, what is it that we’re going to be looking for? Questionmark is wonderful in it provides a wide variety of reporting capabilities. Some of them we rely on very heavily. Some of them we don’t use, because they don’t apply to us. So it’s very important to sit down for us as a department, and then with our senior management as well, and get some clear objectives as to what it is we’re going to be looking for this year before the testing period begins. That way, we have an idea of what data we need to extract once the data is collected. And then there’s always storing that data indefinitely as well. And when you get into data storage you can kind of, if you have the resources and capacity you can store some extraneous data that you may need to come back to later, if you want to use it for a different reason. So I would say communication and create a clear objective is the easiest way not to get caught up in a landslide of data that you don’t need.

Joan: That raises the question of how to determine which data are actionable, which, how you prioritize them.

Keith: Sure. You know, actionable data is very important to us, because if it’s not actionable, then it, you know it’s just collecting numbers. Communication is a big part of that, again, internally. I think that you need to have a discussion with your senior management, whether you work in government, private industry, or otherwise, as to what the goals of your assessment program are. Once you get those goals outlined, you need to really get some sort of resources commitment to say, we can pursue these goals. We can pursue these objectives based on the data that we recover, but we need to know ahead of time that we’re going to have the resources both financially as well as through work hours and so on and so forth, that commitment of, the buy-in of people internally, to go forward with actionable data. Now, to say what is actionable, really, is going to depend on what your organization is looking for. For us, that data is to, is anything that can show us directly what is going to improve student learners and their ability in the classroom. That’s tied directly to our curriculum framework. So we do a lot of curriculum analysis as it relates to our topic structure with an Authoring Manager, which is basically our curriculum standards outline. So for us, that data is very actionable, but we’ve determined ahead of time that it’s going to be the actionable data that we will work on. So that communication, buy-ins from senior management, as well as the commitment of resources to act upon the data that you’ve considered actionable is really a good place to start for anybody. But it’s going to differ drastically based on the needs of your organization.

Joan: Right. So you’ve got the management buy-in. You’ve decided what’s actionable. At that point, how do you effectively communicate about your assessment results?

Keith: We use, again, in large part, we allow year-round access to our custom teacher interface, which allows teachers to kind of go in and get certain reports on their own, to get classroom level data, assessment data based on their students. And then to further that, we provide those students with online resources for classroom development based on analysis of their curriculum standards and their curriculum data. So not only do we give teachers out in the field the actual raw data. We provide them with educational tools to take that data and put it to use in their actual classroom. And we have found pretty empirically when this happens, when teachers use the tools available to them, they generally have better test scores. We also rely heavily around the state on 16 different workforce education coordinators, who are, who have access to Enterprise Manager as administrators to certain reports and documentation. They take that data and then they disseminate it out to our more than 3,000 different teachers that use our system throughout the state. This usually takes place over the summer. And then of course we house data and disseminate it internally, Karen and I do as well, throughout the agency itself. So we have several different tiers of data dissemination once we get that actionable data, once we determine what we want to do with it. To who and where it goes depends on our organizational structure, but we think we have it pretty well organized.

Joan: So how do you use all this data to ensure that student learning keeps pace with what’s happening in the workplace?

Keith: We work directly with industry contacts and industry experts that we have that review our data, that review our curriculum, that review our assessments, and provide a certain amount of validity to them. Karen and myself as well as other program staff within our agency meet so frequently with people in industry. Our assessment data allows us to give them a very in-depth look at what students are learning in relation to their industry. In turn, they can counsel us through workshops, through meetings, through phone calls, discussions, through email discussions that we have, through a number of different avenues by which we communicate with industry professionals to indicate to us how we need to make our product evolve, how we can make our assessments better, how we can make our curriculum more relevant, essentially how we can keep up with what’s actually going on out there in the field. A lot of times, there’s a disconnect between what’s being taught in the classroom and what is actually taking place in, you know in actual industry itself. So this data allows us to exist in a constant state of evolution with our industry partners.

Joan: So which Perception reports do you find to be most useful?

Keith: There are really three reports that are primary reports for us, the first being obviously the score list report, because this is something we can provide to our teachers as basically a grade book report that just gives them a student raw score, provides them some basic information about when the student logged in, how long it took them to take the test, and what their score was. It’s a very simple report, but it’s a report that can provide immediate feedback, that’s incredibly useful.

We also rely heavily on the assessment overview report. The assessment overview report essentially will report based on your topic and subtopic structure as you set it up in your instance of Authoring Manager. We use Authoring Manager topic structure to basically mimic what our curriculum framework standards are. So teachers, when they receive that report, can look at an assessment overview report and not only see how well their students scored, but see how well their particular class scored in relation to our curriculum framework standard, so that they can take that report and basically adjust their lesson planning time to meet their strengths and weaknesses within the curriculum that we provide.

And then finally, we rely heavily also on the test analysis report. And what we use the test analysis report mainly for is to (?12:08) score. That (?12:09) our reliability score is a threshold score that we weigh all of our assessments against that we’ve established internally, and have essentially, know, to the federal government, that if we have tests that perform under this reliability rating, we will either pull those tests or adjust those tests in order to meet that reliability. So we use the score list report, the assessment overview report, and then for our own internal reliability purposes, we rely on heavily on the test analysis report.

Joan: Thank you both for joining me today.

Keith: It’s been a pleasure.

Karen: Thank you, Joan.