Posted by John Kleeman
I’ve been learning how the railway industry uses assessments to maintain competence. As a frequent train passenger, it’s reassuring that the industry and their regulators carefully enforce a safety first mantra. And how railway and rapid transit companies (sometimes with Questionmark software!) use assessments to check competency for rail workers, especially in safety-critical roles.
There are many government and industry bodies that oversee and promote safety including the US Federal Railroad Administration, the UK Railway Safety & Standards Board (RSSB) and the European Railway Agency.
The RSSB have produced a very useful document: Good practice on Competence Review and Assessments. Here is a table summarized from this document that gives pros and cons of different kinds of assessment.
| Type of assessment | What it means | Pros | Cons |
| Observational | Observer watches participant doing normal work | Valid and reliable as it provides first-hand information about performance in real conditionsCaptures information about process and behavior not just outcomes | Risk of a ‘special performance’ as someone behaves differently whilst being observedWill not cover emergencies and other non-routine work Needs good planning |
| Simulation | Participant completes activity which is not real work but replicates real work closely | Provides performance evidence for non-routine workMeasures response to emergencies | Heavy on resourcesNeeds careful planning to be valid and reliable |
| Tests | Formal assessment of knowledge on paper or on screen | Consistent and objectiveGood for assessing technical knowledge Cost effective for large numbers of people |
Requires skill to make valid and reliable |
| Work products | Examining outcomes of work done, e.g. document written or machine serviced | Provides evidence of performance in real work conditions | Need to verify authenticityShows outcome but not route to get there |
| Written reports | Report from participant or colleague describing competence on the job | Provides evidence to support other methods | Need to check authenticityMemory fallible Requires writing skills |
| Oral interview | Conversation where performance is described and questioned | Allows in depth exploration of knowledge and understanding | Relies on skills of assessorsHard to make consistent and objective |
It’s interesting to see the strengths and weaknesses of different kinds of assessments from such a safety-focused industry and to consider how using different assessments together can cover more ground and reduce safety risks.
