Australian College of Rural and Remote Medicine: Creating valid and reliable high-stakes assessments for doctors in remote rural areas


The Australian College of Rural and Remote Medicine (ACRRM) sets and maintains standards for safe General Practice/ Family Medicine in Australia, focusing on the special needs of those living in rural and remote communities. ACRRM provides one of the 2 nationally accredited pathways to Fellowship (Board Registration) in General Practice / Family Medicine in Australia, a process that includes successful completion of a series of high stakes summative assessments.

ACRRM plays a key role in setting the standards for rural and remote medicine in Australia, and it is against this background that ACRRM has extensively employed online delivery tools to support those living and working outside the metropolitan zone.

Distance Learning

“Our main challenge is to encourage doctors to practice long-term in rural areas,” explains ACRRM’s Online Services Manager, Lex Lucas. “To do that we have to be able to provide them with the ongoing education and assessment opportunities they can access, without having to leave their communities to attend.”

ACRRM was an early adopter of distance education and online deployment. For many years the college has used the online learning portal designed and built in-house (known as RRMEO) to deploy education and to track members’ continuing education records.

Virtual Classes

ACRRM members participate in virtual classes that include training for mental health, counselling, diagnosis, and online clinical scenarios using online virtual classrooms.

“Doctors don’t have to travel days to reach one of the major cities,” explains Lucas. “Australia is a huge place and travel chews up time and money. Learning online with ACRRM, doctors do not have to take off a day’s work: they can have dinner with their families, put their children to bed and then attend class.”

Local Assessment Delivery

A key consideration in the development of the ACRRM summative assessment process was to deliver the assessments within each candidate’s local community. This means candidates needn’t take days off work and travel extended distances to attend exams, nor find locums to cover their absence. A unique feature of the ACRRM assessment process is that each exam centre usually holds only one candidate: an exam with 50 candidates can be held simultaneously across 40+ sites, each with 2 invigilators/proctors.

In 2007 ACRRM had a requirement to deliver online high stakes multiple choice question (MCQ) examinations. After an extended search, ACRRM chose Questionmark as an ideal tool to meet these requirements. Since then, ACRRM has successfully developed and delivered three-hour, 125-question MCQ exams run on Questionmark.

Background Using Questionmark Live Browser-Based Authoring

Attending a three-day Questionmark workshop in New Zealand helped Lucas connect with other Questionmark users and gather ideas for future projects using new software features and tools.

“We first felt like we were doing things in a vacuum — by ourselves — and then realised that there’s a whole community of users out there to learn from and bounce ideas off,” recalls Lucas.

During the workshop Lucas became familiar with Questionmark Live, a browser-based authoring tool that enables subject-matter experts to create question sets easily and submit them for inclusion in Questionmark assessments. He says Questionmark Live will help ACRRM move to a more integrated system across the exam writing, editing, delivery and marking process.

“Rural doctors, academics, and editors will all be able to log into Questionmark Live, review changes, refine and manage question writing without the expense or inconvenience that face-to-face meetings would require,” Lucas explains. “Questionmark Live will help us improve the quality of our questions by tracking versions and revisions, thereby strengthening the integrity of our assessment process.”