Case highlights
- With the Questionmark platform, the CareerTech Testing Center (CTTC) was able to deliver and streamline three distinct certification processes to disparate customer types without compromising on security.
- With the use of Questionmark reports, the CTTC could deliver extensive data insights and support its customers in making data-driven decisions.
- By implementing tools like Item Analysis, the CTTC could enhance the fairness and validity of certifications both internally and externally.
- Using tools like the Scheduling Management System, the CTTC was able to streamline and speed up administrative tasks for certification setup.
- With access control and data compartmentalization, the CTTC could deliver certification programs that kept customer data secure.
- With the use of Questionmark reports, the CTTC could deliver extensive data insights and support its customers in making data-driven decisions.
Tools used
- Item Analysis
- Reporting
- Integrations
- Proctoring
- Locked Down Browser
- Scheduling Management System
Scenario
The Oklahoma Department of Career and Technology Education aims to develop a world-class workforce for Oklahoma employers and prepare Oklahomans to succeed in the workplace, in education, and in life. It assures standards of excellence for a comprehensive statewide system of career and technology education. That system offers programs and services in 29 technology center districts operating on 60 campuses, 391 comprehensive school districts, 17 Skills Centers campuses (that include juvenile facilities) and 32 adult basic education providers.
The CareerTech Testing Center is a division within the agency. Since 1985, the CTTC has worked directly with instructors, industry representatives and credentialing entities to identify and develop more than 100 certification assessments and assessment preparation materials which align with recognized industry standards. The CTTC also collaborates with subject-matter experts in a given field to provide elevated insight and guidance, using that analysis as a blueprint for any associated assessments. These standards and learning objectives then define the goals and objectives for a program and any associated assessments that follow.
The CTTC also uses a technology center network of more than 50 locations to provide test sites to its 15 partnering agencies and divisions.
The move to Questionmark
The CTTC returned to using the Questionmark assessment platform in 2019 after some initial trials in previous years with other testing platforms. The decision to transition to Questionmark’s platform was primarily driven by the need for support in three key areas: robust reporting, facilitating the CTTC’s internal certifications program, and enabling the delivery of distinct certification processes to the CTTC’s three diverse customer bases.
Tailored site solutions
One of the main benefits to the CTTC was Questionmark’s ability to help it deliver a customized experience and process to each of its various customers.
“Our customer base consists of the test sites that provide our in-state competency testing; our out-of-state test sites that utilize our certification assessments and testing platform for their state-mandated testing; and our agency partners that needed us to create and/or host high-stakes licensure exams. Each of these areas have very different processes … with the help of Questionmark, we had the ability to set up three different types of processes to handle our diverse customer base.”
Jennifer Lathem-Palacio, Assessment Manager, CareerTech Testing Center
With Questionmark’s support, the CTTC was able to provide customized processes for each type of testing site in keeping with customer needs.
For example, the CTTC utilized the Questionmark platform for in-state competency testing requirements by facilitating proctors to be able to submit student rosters and test details. Multiple rosters could be submitted by sites offering various programs that the CTTC then uses to create student profiles, provide login information and schedule tests — all within the secure and streamlined Questionmark platform.
For out-of-state competency test requirements, the CTTC leverages the Questionmark platform to gather information from registration questions on tests and from that deliver information like bulk score reports to customers.
To manage the huge administrative task of facilitating high-stakes licensure testing (including 50 test sites across Oklahoma and 15 agency partners and divisions), Questionmark allows the CTTC to use testing centers and system integrations to easily and securely add and schedule users based on their unique registration data provided by the customer.
User and data compartmentalization are crucial for success across all these setups, and the CTTC accomplishes this by using Questionmark Group and Scheduling features to ensure appropriate and secure access control.
“Within all of these setups, we had to compartmentalize users and data so that each site only had access to certain information. We utilized groups to structure the various rosters on the competency side of the house for both in/out-of-state customers. We also used groups on the high-stakes side to control access to the various testing that is contracted and to keep candidate information secure (candidates can choose to move around between test sites, but we utilize a formula for user creation that allows us to keep all results tied to each candidate).”
Jennifer Lathem-Palacio, Assessment Manager, CareerTech Testing Center
This ability to confidently categorize data in a way that’s secure allows the CTTC to make full use of registration and exam results data for a more streamlined process that effectively delivers certifications in keeping with customer needs.
Better reporting and item analysis capabilities
The ability to provide the best service possible relies on the CTTC having access to the necessary tools and this is especially true for reporting and analytics.
“The decision to work with Questionmark was heavily influenced by its various reporting capabilities. With other platforms, in-depth analytics were not available, and that was something we needed in order to provide the best feedback to our stakeholders to determine item revision and development. Using the Questionmark platform meant we could use item analysis, export to ASCII, coaching reports and group analysis throughout all of our testing avenues.”
Jennifer Lathem-Palacio, Assessment Manager, CareerTech Testing Center
Ensuring question quality with item analysis
With access to Questionmark’s item analysis capabilities, for example, the CTTC can provide a comprehensive evaluation of test items, identifying their difficulty levels, discriminatory power and overall effectiveness for both internal and external certifications:
“We heavily utilize the item analysis to review our in-house certification assessments and to provide feedback to our other partners that we provide licensure testing services to.”
Jennifer Lathem-Palacio, Assessment Manager, CareerTech Testing Center
In doing so, the CTTC gains valuable insights into the effectiveness, fairness, and quality of its certification exams. This meticulous approach not only enables the detection of ambiguous or misleading questions but also addresses difficulty and discrimination challenges by identifying items that accurately measure the intended knowledge or skills, effectively differentiating between high-performing and low-performing candidates.
This valuable information plays a crucial role in enhancing the reliability and validity of both the CTTC’s internal and external certification processes, ensuring that the assessments taken provide accurate and meaningful results.
Actionable insights from detailed reporting
“We use the reporting features — including coaching and group analysis reports — to provide detailed reports to the various agencies that rely heavily on registration data that must be collected from the candidate at the time of testing.”
Jennifer Lathem-Palacio, Assessment Manager, CareerTech Testing Center
The availability of coaching and group analysis reports is particularly useful for the CTTC, which is especially concerned with delivering the necessary levels of insight to its various customer types.
Coaching reports, for example, provide customers with detailed insights into individual performance, enabling targeted interventions and personalized support to enhance test-taker outcomes. Equally, group analysis reports provide a broader perspective by analyzing aggregated data, identifying trends and informing decision-making regarding curriculum development and instructional strategies. Better still, with Questionmark reporting, many of these data points are viewed over time to give a more accurate picture of each item and test, as well as from a top-level view that also provides an automatic prompt for drilling down into a more detailed analysis should it become necessary.
Ultimately, the combination of detailed item analysis insights and the comprehensive reporting features offered by the Questionmark platform allow the CTTC to enhance its certification exams and deliver a truly data-driven experience to its extensive customer base as well as to its own workforce.
A push for online proctoring
In the current landscape, it’s not surprising that the desire for more online methods of assessing candidates has also given rise to the demand for robust proctoring solutions and this was another need that the CTTC is keen to address.
“We do have a number of partnering agencies and divisions that also prefer us to provide them with an online proctoring solution for testing. We have around 22 assessments that we offer which includes this service. We utilize Examity through Questionmark to provide a portal that allows these candidates to purchase their exams, schedule the, and take the exams online.”
Jennifer Lathem-Palacio, Assessment Manager, CareerTech Testing Center
The CTTC, needing a seamless online proctoring solution, found the integration of Questionmark with Examity particularly helpful when delivering on customer requirements.
In leveraging this integration, the CTTC confidently offers its customers a robust and secure proctoring solution that not only leverages AI technology for excellent test fraud detection but also allows for a high degree of customization. This focus on test integrity and commitment to maintaining the credibility of assessments is a core challenge of any certification program and one for which the CTTC wants to provide strong solutions that also meet customer expectations.
Digital badging for validated certifications
Digital badging has become an increasingly common tool for enhancing certifications and one the CTTC is keen to offer. Not only can digital badging provide its customers with a powerful means for recognizing and validating achievements, but it also provides customers with a paperless way of sharing their success.
The CTTC used Credly in particular for providing digital badges for its various certification programs.
“We are also now issuing badges through Credly. We know that Credly has an interface with our testing platform, but the badges we are offering come from Oklahoma Department of Career and Technology Education’s approved assessment list. This list covers credentials from many different vendors. We are utilizing our Information Management Data System to gather the data needed to issue badges through Credly.”
Jennifer Lathem-Palacio, Assessment Manager, CareerTech Testing Center
Ultimately, embracing digital badging was something the CTTC wanted to deliver to its customers; and its work with Credly allowed this to be done successfully. You can find all the digital badges offered by the CTTC here.
The plan ahead
As we continue to work with the CTTC on delivering the very best service to its large and differing customer base, we asked Jennifer what the CTTC’s plans were for future development.
“We hope to add the new Text-to-Speech option to our offerings next fiscal year. We are always looking at providing solutions to answer the request for accommodations and this one will work with the Secure Browser functionality that we utilize on the high stakes avenue too … we are also looking at possibly adding observational assessments to our offerings, but we are in the early stages of that process.”
Jennifer Lathem-Palacio, Assessment Manager, CareerTech Testing Center
Results
By switching to Questionmark’s online assessment platform, the CTTC was able to achieve the following:
- Deliver three distinct testing processes for customer bases with very different needs. This includes in-state competency testing, out-of-state test sites for certification assessments, and agency partners that required high-stakes licensure exams.
- Speed up processes and successfully compartmentalize customer data securely with tools like the Questionmark Scheduling Management System.
- Elevate exam validity and fairness with Questionmark tools and features like item analysis and integrations like proctoring with Examity.
- With the use of Questionmark’s extensive reporting features, including the group analysis and Coaching report, attain in-depth insights not only for numerous customers but also for CTTC’s own internal certifications.
- Look at other options for using the Questionmark platform for future endeavors, including the use of observational assessments and the Text-to Speech tool.
- Add digital badging to its services with Credly.
- Put security at the forefront with proctoring, locked-down browsers and access control functionality.
To discover how Questionmark could transform your learning and development process, talk to us today.
