Case Study: Using Diagnostic Assessments to Improve a Government Agency’s Workforce

Background

The Aurelius Group (TAG) provides Federal acquisition, human capital, and technology consulting to private industry, federal agencies, and the U.S. Department of Defense. The company, a verified Service Disabled Veteran Owned Small Business (SDVOSB), creates progressive, end-to-end human capital solutions for its clients. These initiatives include customized employee assessment programs.

Scenario

Faced with an expanding workload, inexperienced employees and increasingly scarce resources, a large Federal agency needed to identify and close proficiency gaps in their acquisition workforce.
 
The agency’s approximately 8,000 FAC-Certified employees had been accustomed to taking self-assessments, but the agency wanted a workforce improvement program that would include objective, defensible, competency-based assessments. The client required a flexible program that would minimize workforce burden, technical complexity and cost while ensuring that employees were optimizing their development and training efforts.

Solution

Fortunately, the Federal Acquisition Institute’s (FAI) Federal Acquisition Certification (FAC) Program provided a framework for focusing on key positions and competencies. Based on this framework TAG designed an assessment program to enhance leadership’s awareness of the strengths and weaknesses of the FAC population and to improve the efficiency with which training and workforce development dollars were spent.           

 

In creating a program to meet the agency’s needs, the firm combined its three core competencies—acquisition expertise, strategic human capital projects, and technology consulting — to create a unique assessment solution. At the program’s outset TAG had two primary goals. First, the team developed the business rules that would allow the addition of assessments to seamlessly integrate with their client’s current workforce development framework. Second, TAG needed to evaluate and select a flexible technology platform capable of supporting those business rules and any changes that might be made to them in the future.

 

The project team analyzed about 50 testing and assessment systems, from which Questionmark emerged as the most comprehensive platform for measuring employee competencies.

 

“We pulled Questionmark off the shelf and were able to start giving assessments a few months later in a fairly customized way,” recalls Aurelius Group Chief Information Officer Justin Devine. “We’ve been able to adapt our assessments to changes in the program, and we’ve worked with Questionmark to create a system that matches our client’s needs. Objective assessments provide clear, measurable information about people’s skill levels and highlight areas where they need to improve their knowledge and skills. Test results also help us monitor the success of the program and continuously fine-tune it.”

Assessments Reflecting the Work Environment

Questionmark assessments provide annual or biannual snapshots that show how much employees know about the complex bodies of knowledge their work requires and identify competency gaps that can be addressed through further learning. The assessments use randomized questions from large item banks pertaining to three different disciplines: Federal Acquisition Certification (FAC) for Contracting, Contracting Officer’s Representative, and Program Project Managers. After completing an assessment, each participants immediately receives targeted training recommendations that feed into his or her Individual Development Plan (IDP).

 
In addition to helping individual employees understand their current knowledge levels and how they can improve, these diagnostic assessments provide an enterprise view of the strengths and weaknesses of the entire workforce.

Assessments are administered using Questionmark via virtual servers in the Cloud. This makes them available to registered users anywhere on the internet. Employees may take them from anywhere they wish. Since they need to look up information on the job, they are allowed to do the same when they take these “open-book” assessments.

Reporting and Analytics

Careful attention to reporting and analytics help the project team provide data-driven management, planning, and decision-making. 
“We evolve the content of the program based on data gleaned from tests and on what analytics tell us about the test results,” explains Justin. “In addition to what we gain from the technology, we do deep reviews of our competency modeling and competency assessment programs to make sure we stay on the right track.”

 
Analysis of test results also holds a lot of value for employees, who learn from Questionmark’s Coaching Reports about their strengths and weaknesses across competencies relevant to job performance. Test results play an important role in measuring employees’ current levels of expertise and directing them to appropriate learning and development efforts.
 
“We have produced thousands of Coaching Reports and use the outcomes to point people to further training,” Justin says. “We encourage the employees to sit down with their supervisors and use the report as input for their IDPs. Participants get targeted training recommendations that lead them to the highest-value learning opportunities.”
 
The Question Statistics Report provides a quick, easy way to determine a test item’s P value or level of difficulty. The statistician andpsychometricians run their own ASCII reports to drill down further, using item response theory to evaluate results.

In the future, the team hopes to use the Questionmark Gap Report to compare groups of participants with each other to determine differences achieved at the question, topic or assessment level — a means of tracking and analyzing performance over time.  Also in view is the use of Questionmark’s Results API, which uses OData to collect results and enables dynamic, customized reporting via business intelligence tools. 
“We’ve been developing some proof of concept dashboards using OData and Microsoft’s PowerBI suite and we’re really excited about the early results.  We’ve decided to make reporting and analytics based on assessment results a core competency for our firm.”

Getting Credit

 The program includes Continuous Learning Points (CLPs) for assessment participation, providing an incentive for people to be aware of and build upon their knowledge, skills and abilities. TAG has created a custom extension to Questionmark that automatically generates completion certificates and sends them as email attachments after people successfully complete assessments. 


“This gives people tangible proof that they use to claim the continuous learning points they earn to maintain their FAC certifications,” explains Justin. “It provides evidence that they did indeed complete the assessment. Also, the process of taking the assessment is valuable in and of itself, since responding to questions has been shown to enhance learning.”

Results

 Assessments have contributed greatly to the success of the workforce improvement program, which reveals the aggregate strengths and weaknesses of the workforce and enables the client to direct resources to high-value development opportunities. Trend data gleaned from the assessments demonstrates decline and improvement over time and provides objective support for training resource requests.

At the individual level, the program has provided thousands of participants with coaching reports, improved their self-awareness, helped shape their individual development plans (IDPs) and resulted in more effective learning choices.
 
 “Questionmark has worked well for us,” Justin concludes. “We’ve used a lot of the capabilities, some of which we’ve relied upon to do things that are somewhat unique. The technology has been flexible enough to let us change things without having to ask a developer to add new features. It’s been a cost- effective and reliable way to identify the strengths and weaknesses of the workforce while giving participants credit for their work and pointing them toward appropriate training.”