• Home
  • Book Review
  • Criterion-Referenced Test Development: Book Review
  • Criteri...

Posted by John Kleeman, Founder and Executive Director

This is the second in a series of Questionmark book review blogs. Our first review was about Eric Shepherd and Joan Phaup’s book, “Talent Transformation” about how assessments help make people decisions. This review steps back a little in time and looks at one of the great books on testing – Criterion-Referenced Test Development by Sharon Shrock and William Coscarelli.

Challenges the book deals with

Frequently, in the world of work, we need to know if someone has the competence to do something. Can a bank salesperson speak to a customer to sell a loan? Is an employee safe to go into a room containing hazards? Can an engineer safely be sent to fix machinery? Tests like this which measure people against a benchmark or a specific level of competence are called criterion-referenced tests.

Creating a test which measures competence in a trustable way is not easy. The key to this book’s value and longevity is that it is aimed at the ordinary trainer or other person without specialist assessment training who needs to do this in the corporate environment. The book was first written over 20 years ago and the current third edition was produced in 2007. But the challenge remains now as then – how does an ordinary trainer or HR person produce a test which can be used in a trustworthy way in the corporate world?

What the book covers

Many books written by testing experts are difficult to access without mathematical and statistical knowledge. The great thing about this book and the reason it withstands the test of time is that it’s written by experts but it’s accessible by ordinary people. The book walks through the rationale for making tests for corporate training and explains concepts in detail and then explains the process. There are sections on:

  • Documentation required for legal defensibility
  • Job analysis
  • Test objectives
  • Cognitive tests and performance tests
  • Creating items
  • How many items should there be in a test
  • Creating rating instruments (Likert Scales, etc.)
  • Piloting tests
  • Setting cut-off scores (pass scores)
  • Reliability of tests
  • How to report scores

The book also has a section on legal issues, mostly with a US focus, covering the documentation required for tests to make them legally defensible for use in employee selection in the US and how to validate tests appropriately.

And much more …

You can see a full list of contents on the publisher website HERE.

The book covers a lot of detail – it is 450 pages long and there is useful, “meaty” content on most areas. For example, the chapter on setting cut-off scores is 23 pages and describes three methods of setting cut-off or pass scores: Informed Judgement, Angoff and Contrasting Groups. It explains how to use them and gives advantages and disadvantages and other commentary.

How the book might help

This is the one book on testing which I refer to most often, it’s particularly useful when I need to remind myself on an important concept or when I want to see their view on a particular method or issue. When I had to move to work at home in a hurry earlier in the year due to COVID-19, this is the one book I missed, and I’m very pleased to have it back with me. 

Shrock and Coscarelli have done a lot of consulting and practical work in testing over the years, and they are probably most famous for their suggestion that there are two practical things in testing that they’ve found to improve testing in most organizations (you can see their paper on this HERE) . One of these is to use the Angoff method to work out a cut score (and not just to pick an arbitrary number, e.g. 70%). The other is to write questions above knowledge, here is a quote from the book:

“In general, the single most useful improvement you can make in test development is to write items above the memorization level”

The book is a classic, so there are some recent developments not covered, but it’s nevertheless the best book on testing I’ve ever read, and I strongly recommend it to anyone working in the world of testing.

If you like to connect more on this book or learn more about Questionmark, please contact us.