Trucking Firm Administers Questionmark Assessments using Satellite Communications Systems
With over 3500 drivers, more than 2600 trucks and six driving schools, Salt Lake City-based C.R. England must constantly train new drivers. The trucking industry faces a very high turnover rate – a whopping 140 percent per year on average. Using experienced drivers to train apprentices, the company constantly refreshes its staff of “over the road trainers” using a pool of 600. Each trainer takes an apprentice driver out for a period of 4-6 weeks of on-the-road training.
As with any company orienting new workers, C.R. England looks to evaluate how these apprentices are performing. But how do you do that in a rolling classroom? If, for example, the trainer/truckers administer paper-based tests, how does the company obtain the results on a timely basis and be assured of their validity?
According to Camille Price, instructional design manager at the company, paper-based assessments are very easy to use and inexpensive, but they proved to be an ineffective assessment approach.
“They would frequently get lost; some people would just check the boxes and wouldn’t actually complete the test, and by the time we got the data back, the prime opportunity for remediation was gone.” she said. “It was very labor intensive because if we were going to actually analyze any of that data, we would have to collect it all and have somebody key those results in manually.”
If the company couldn’t evaluate results quickly then it couldn’t evaluate the progress an apprentice was making while out on a truck for 4 to 6 weeks,
Price evaluated using a system that would test students over the telephone. In a variation of the typical menu driven voice mail system, this approach would give students choices over the phone and allow them to make selections by punching the appropriate key. While the concept was viable and would provide current data, the cost of initiating such a system proved to be extremely expensive and, therefore, not desirable. The system was also cumbersome for the driver to use and only universally accessible to drivers willing to pay roaming charges on their cell phones.
Just about every commercial tractor trailer in the United States is fitted with a small satellite bubble mounted to the top of the vehicle. The C.R. England fleet of more than 2,600 tractors is equipped with a two-way satellite communications system. This enables dispatchers to have instantaneous contact with every driver and know the exact location of each tractor.
The CRE training management team realized that the company could use the same unit, which has a keypad with limited functions and small built-in screen, to administer their tests. While the drivers are gaining on-the-road experience, those tests ensure the apprentices also understand the rules and regulations for driving, docking protocols, company policies, and the proper handling of hazardous materials. All of this information, which they learn during their initial 4-6 week training period before becoming solo qualified, can be tested using basic multiple choice questions.
While the satellite system worked well as a communications device, C.R. England still needed an assessment management system to deploy the testing process in an effective manner. After evaluating and rejecting several learning management systems, which handled item analysis poorly and provided features she didn’t need, Price decided to use Questionmark to create and then analyze the data from these tests.
Using middleware created by Questionmark partner company Alpine Media to make it possible for the communications system and Questionmark to talk to each other, C.R. England established a system in which sets of questions are created in Questionmark and then printed in hard copy format for distribution to the apprentice drivers. Delivering the questions over the satellite network would have been prohibitively expensive. Instead, drivers get one of five different quiz booklets, depending upon the fourth digit in their user ID number. Randomizing the tests in this manner ensures better security. At the appropriate time, each driver takes a test, entering the answers into the satellite unit by selecting the appropriate letter choice on the small keyboard.
One of the newer features of Questionmark, QMWISe which acts as an application program interface (API), creates new users in C.R. England’s database and then assigns them the appropriate tests. The interface feature enables the company to use the custom programming it created on its existing IBM AS/400 computer.
Once the student enters his or her answers, these are transmitted via satellite, creating an SQL database entry. The middleware validates that the person sending the answers is indeed a registered user, polling the Questionmark participant tables. The middleware scores the assessment and then sends back a confirmation about whether the student passed or failed. All item choice distribution data along with the assessment score and number of attempts is then sent to the results tables in Questionmark.
If apprentices are laid over at a CRE terminal with a computer lab, they can also take the assessments over the company Intranet.
Another fundamental element of the C.R. England training protocol consists of what the company refers to as “check calls.” These are weekly, live telephone interviews that company training coordinators make. Check calls are essentially fail-safe measures to ensure that new drivers have mastered what they need to know and are ready to take the next step. Instead of asking simple multiple choice questions, the training coordinators pose open-ended questions that elicit more detailed responses.
Questionmark assists this effort also. The training coordinators log into Questionmark as if they are the student, using Questionmark's monitoring feature. Monitoring verifies the identity of the person taking the test and prevents a student or anybody else from entering evaluations without authorization. Questionmark then prompts the coordinators with a script of open ended questions. They read these to the student over the phone and render an evaluation based upon student answers, which is logged into the program.
Training constantly moving targets presented a real challenge for C.R. England. The company not only must closely monitor the results from hundreds of new truck drivers per year, but also discover the areas in which they need additional instruction – often called gap analysis. One of the things that drew Camille Price to Questionmark is its capability for item analysis to aid that process.
“What was really cool was Questionmark’s item analysis that reveals our knowledge gaps,” she said. “Questionmark's statistical analysis of the questions we were asking highlighted those areas and, thereby, makes our training more targeted.”
For example, Price discovered through item analysis that apprentice drivers had trouble understanding how to fill out their daily logs, a critical operation for truckers. Now C.R. England is teaching this differently. One of the great strengths of Questionmark is that by analyzing results, companies can actually improve their training.
Another aspect of Questionmark that sold Price was the ability to carry unlimited user records in the results table. “I might train 5,000 drivers a year, to keep a driver pool of 3500 drivers,” she commented. Obviously the kind of flexibility that Questionmark offers makes it possible for C.R. England to train a large and ever changing pool of workers.