When creating tests that define levels of competency as they relate to performance, it’s essential to use a reliable method for establishing defensible pass/fail scores. One of these is the Angoff Method, which uses a focus-group approach for this process. The Angoff scoringmethod has a strong track record and is widely accepted by testing professionals and courts as one of the best ways of calculating the cut-off score for tests.
The Angoff method calculates the cut-off mark in relation to a defined standard (absolute) as opposed to how they perform in relation to their peers (relative). The cut-off score then divides candidates into two groups, those below the cut-off and those above the cut-off which may indicate a pass or a fail.
The difference between the Angoff scoring method and other ways of calculating test cut-off marks us that it involves a judgement being made of the difficulty of items in the exam (test-centered) as opposed to how candidates perform (examinee-centered), and is widely used to set the required pass standard in high stakes examinations.
There are several variations of the Angoff method, but this article should give you an understanding of the process adopted by most organizations.
How is Angoff calculated?
In order to calculate Angoff, subject-matter experts (SMEs) review each test question and then predict how many borderlinecandidates would answer the item correctly. Before making a judgment, the experts must all agree on the definition of a ‘borderline’ candidate in order for the Angoff cut-off score to be reliable. The judges must be experts in order to have the understanding of what makes a candidate competent enough to pass the exam.
The experts’ scores should be the same or fall within a close, defined range (often 10%). The average of the judges’ predictions for test questions, also referred to as the ‘predicted difficulty’ is used to calculate the passing percentage (cut-off score) for the test.
Should you use Angoff in your organization?
Angoff is a well-established method of standard setting. It’s most commonly used in high stakes exams and is most reliable when supported by another standard setting method.
Advantages of using the Angoff method
- Holds up in court
- Reflects the difficulty of the exam content
- Simple when done properly
- Can be reused across other exams with the same context
Disadvantages of using the Angoff method
- Considered more reliable if backed up by a criterion-referenced method
- Long process that can lead to judges becoming fatigued and impatient
- Judges must be confident and consistent with their definition of a borderline candidate
- Large sample size of judges with a good range of different ages, genders, ethnicities and seniority can be difficult to achieve.
Basing cut-off scores on empirical data instead of choosing arbitrary passing scores helps test developers produce legally defensible tests that meet the Standards for Educational and Psychological Testing. The Angoff Method offers a practical way to achieve this.