Questionmark Conference 2017: Innovate. Inspire. Succeed.


  Maximize your assessment value

Are you prepared for the next generation of assessment management?

 Harness the power of your results

Are you getting the data and insight you need to make informed decisions?

 Innovate with us on future assessment technologies

What are your measurement needs today -- what will they be tomorrow?


Pre-Conference Workshop

Advanced Test Item Writing Workshop – Learn how to test more than just knowledge

  • Date & Time: Tuesday, March 21 -- 9:00 a.m. - 4:00 p.m. (Pacific Time)
  • Location: Inn and Spa at Loretto, Santa Fe, New Mexico
  • Instructor: Jim Parry, Owner and Chief Executive Manager of Compass Consultants, LLC
  • *Registration Fee: $495 for conference participants

Click to Register
Boot Camp & Questionmark Conference 2017



Writing test items is difficult, but trying to make them check more than knowledge is a huge challenge.  All tests should be well developed, and testing practices should benefit participants and trainers alike.  There is extensive evidence documenting the effectiveness of well-constructed tests in relation to supporting the validity of the test. The improper use of tests, however, can cause considerable harm to test takers and others affected by test-based decisions.

This fast-paced, high-powered workshop will review the basics of testing and provide hands-on practice to help you turn low-complexity, knowledge-based test items into higher-complexity, performance-based items following Bloom's Taxonomy and Gagne's Nine Events of Instruction. It will include a demonstration of several features available within Questionmark OnDemand that can be used to help develop higher-complexity test items.

The workshop will cover these topics:

  • Criterion-Referenced vs. Norm-Referenced Test
  • Reliability and Validity
  • The Real Meaning of a Test Score
  • Measurement Errors
  • Test Defensibility
  • Test Development & Revision
  • Design of a Test Plan and Supporting Documentation
  • Test Item Construction
  • Test Construction
  • Scales, Norms and Score Comparability
  • Establishing Cut (Passing) Scores
  • Levels of Complexity
  • Matching Complexity to Cognitive Performance
  • Item Writing Practice
  • Frequency Distributions
  • Test Analysis and Decision Making
  • Test Evaluation (Common Statistics)
  • Ensuring Test Quality

To participate in this workshop, you will need the following:

  • An open mind to think outside the traditional box of multiple-choice, true/false testing
  • A laptop/tablet computer for item writing practice (helpful but not required)

Learning objectives:

  • Describe the difference between criterion-referenced and norm-referenced tests
  • Describe the importance of reliability and validity in testing
  • Review the factors that affect achieved test scores
  • State the key factors to test defensibility
  • Design a test plan
  • Describe various methods used to establish test cut (passing) scores
  • Design test items that match complexity to difficulty
  • Review the importance of test analysis

Recommended Experience Level:

  • Intermediate to advanced. Beginners will also benefit from the information presented but should have had some exposure to test item development.

About the Instructor:

Jim Parry, a long-time presenter at the Questionmark Conference, is the owner and chief executive manager of Compass Consultants, LLC — specializing in training and testing solutions. He is a Certified Performance Technologist (CPT) and holds a master's of education degree.  Jim has over 30 years' experience in training and assessment design, development and presentation.  Upon leaving active duty after over 22 years with the US Coast Guard (USCG), he spent more than 5 years performing crane and hoist inspections for OSHA safety compliance, conducting accident investigations and training operators. After that he was vice president of training and marketing at a major maritime training institute.

Most recently, Jim served his country again as a civil service employee as the test development and e-testing manager at USCG Training Center Yorktown, Virginia, where he managed the design and development or update of several high visibility courses.  As the recognized expert, Mr. Parry developed standard operating procedures establishing testing policies and processes for use throughout the Coast Guard. Additionally, Jim served as system manager for the USCG’s Questionmark test development and management system. He has presented several papers at the International Society for Performance Improvement (ISPI) annual conference and the Coast Guard Human Performance Technology (HPT) conference in addition to the Questionmark Conference. Jim received several awards for his superior performance while employed by the US Coast Guard as well as the "Getting Results" award from Questionmark.

For additional information about Compass Consultants, LLC and Mr. Parry, please visit

Pre-Conference Workshop