Creating Reliable High-stakes Exams at the University of Connecticut Health Center
Founded in 1961, the University of Connecticut Health Center (UCHC) is composed of the School of Medicine, School of Dental Medicine, John Dempsey Hospital, the UConn Medical Group and UConn Health Partners and University Dentists. The Health Center provides health care education in an environment of exemplary patient care, research and public service. The Health Center's 206-acre campus is in the historic community of Farmington. With approximately 5,000 employees, the Health Center is Connecticut's 16th largest employer.
Each year, about 320 students work toward their medical doctor's degree and 160 toward their doctor of medical dentistry degree. In the years since UCHC graduated its first students in 1972, 1,329 men and women have received their D.M.D. degree; 2,819 their M.D. degree.
UCHC uses Questionmark to create and deliver high-stakes exams to medical and dental students as well as lower-stakes tests and faculty evaluations. Questionmark Secure is used for delivering 10 to 15 high-stakes exams every semester, each consisting of up to 150 questions. First and second year students take exams in three to four courses a semester. Courses such as Organ systems 1, 2, 3 and Human Biology all require students to take a four to five-hour comprehensive assessment that can include multiple choice, drop down, essay, multi-part essays.
For histology testing, UCHC Academic Computing Specialist Joshua Simon integrated a third- party Flash-based web application that can replicate the functions of a microscope into Questionmark. “Faculty wanted to have students look at a slide and answer questions based on what they saw,” explains Simon. To answer questions based on the slide, students click on a microscope icon in the test and are able to zoom in and out and move the slide from side to side. The images are actual glass slides, some of which are scanned up to 40x. The Flash file is then embedded into a multiple choice question inside the Questionmark exam. “It’s really kind of awesome,” Simon adds. “It’s as clear as if they were actually sitting in front of a microscope.”
High-stakes assessments require questions to be reliable and defensible. Questions are created by faculty members who email or drop off their questions to an education support office, where they are uploaded to Questionmark Authoring Manager. After each exam, Questionmark's Item Analysis Report is used to help determine the quality of each question. A faculty committee reviews the reports, looking for faulty questions. They then decide whether to improve a question or simply drop the question and give students the credit for it. Simon has used the Questionmark data to create his own custom reports that allow him to easily sort data by discriminating value. He has created several other custom reports from the Questionmark data as well as from running queries directly against the database (SQL).
Students must evaluate the faculty member (s) who lectures each week. “Students authenticate through a custom website, which logs them into Questionmark and lets them evaluate their faculty,” Simon explains. The survey system imports the necessary data from the university’s learning management system and sends out emails, tallies up the surveys and creates reports. “The surveys themselves are anonymous but we’re able to see who actually went in and filled out their survey requirement for the week,” Simon adds.
UCHC has successfully customized Questionmark to meet their particular needs. “We’ve come up with a system that works really well for us and we’re happy with our workflow,” Simon comments. “We’re going to make the switch to Questionmark version 5—but other than that I don’t think we’ll be changing much of what we’re already doing.”