OnDemand Testing Saves Medical School Time and Money

Background

The Florida State University College of Medicine offers the Doctor of Medicine (M.D.) degree for physicians and a Ph.D. in Biomedical Sciences. Its faculty includes scientific researchers and practicing physicians, as well as researchers who focus on sociological and humanities issues in medical services industries.
 
With its main campus in Tallahassee, where students complete their first two years, the college also has six regional campuses plus an optional Rural Track medical education program, which places third-year students in small-town practice settings in Northwest Florida.

Scenario

The college was spending about $80,000 annually on pencil-and-paper exams that included full-color photographs, when FSU Professor of Biomedical Sciences and Associate Dean Lynn Romrell joined the faculty in 2008.
 
“Everything skyrocketed when you had to produce full-color pictures,” Romrell recalls. “I had come from the University of Florida College of Medicine and had used computer-based tests since the mid 90’s, so we began looking at different types of testing software for FSU and concluded that Questionmark was probably the best one available.  One of our goals was to create exams similar to those given by the National Board of Medical Examiners (NBME), which produces the U.S. licensing exams, as well as subject exams given throughout the country. It was important to represent the curriculum in various disciplines and make sure we didn’t ask questions that would be emphasized on one campus and not another.”

Getting Started

The college installed Questionmark in-house and started testing first- and second-year students.
 
“We started out just with quizzes in various courses, since the faculty didn’t want to risk high-stakes exams,” says Romrell. “By using quizzes we demonstrated that it could be successful, and it took about a year to win everybody over. Now, essentially all exams are given online using Questionmark.”
 
Most questions are multiple choice, often displaying photos and other images, similar to the NBME Subject Exams. Faculty members create questions in Microsoft Word and then staff in the Office of Medical Education (OME) assemble the assessments using the Questionmark tools.

Switching to Questionmark OnDemand

Although the idea of online testing caught on well, delivery problems threatened to derail the program. Disruptions caused by server and/or network problems prompted the college to switch to Questionmark OnDemand, a software-as-a-service solution that delivers assessments via the Internet. The OnDemand system’s SSAE 16-audited data center provides a fully secure environment with more than 99% uptime. This switch has freed the college from having to update software and configure and maintain hardware.  
 
“When you have control of your own software, you have to update everything yourself, but now we just take advantage of the updates as they occur,” explains Romrell. “And if there are any technical problems, we know that the Questionmark people are paying close attention to them and fixing them quickly. We don’t have to worry locally. Before, we had to have local IT staff take time from their other responsibilities to make sure everything is working okay on our in-house server.”

Test Security

Quizzes and tests are administered on students’ laptops via a wireless network. Students take tests in classrooms, with specially trained proctors present. Questionmark Secure is installed on students’ laptops to prevent task switching. Other measures for discouraging cheating include assigning students to widely-spaced desks and randomizing test answers. On typical testing days, students take the examination in three different classrooms.

Using Post-Exam Reports

After completing an exam, students can schedule a time to view the Questionmark Coaching Report, with which they can review questions, see what they got right, see what they got wrong and determine where to focus their energies.  Faculty members use the Item Analysis Report to see which of their questions are effective, and OME staff provide standard reports to give the faculty feedback on the quality of the examinations.

Saving Time and Money

The switch to online testing has saved the college money by eliminating the need to print full-color photos and publish exam books. And now, having delegated test delivery and technical troubleshooting to Questionmark, college IT staff have more time to focus on their core responsibilities.