Posted by Jim Farrell
On Monday, Austin Fossey takes over as Questionmark’s Product Owner of Reporting and Analytics – the person in charge of our reporting tools. Austin will be working with our customers and product development teams to make sure our reporting and analytics teams solve real business problems.
The other day I asked Austin some questions to help you see what a tremendous addition he will be to the Questionmark team.
What has been your career track so far?
Most recently, I have been developing assessment and value-added reporting systems at the American Institutes for Research. Before that, I spent three years doing test development and psychometric work for a certification company in the construction industry. I also spent a year at the Independent Evaluation Group at the World Bank, which does program evaluation.
What are some of your career highlights?
I was very excited to be a member on a team that developed data training for K-12 educators. This training built upon previous programs I had implemented, and it really took data analysis to a higher level. The training focused on using data to drive instructional decisions at the class, school, and district level. What I liked about this project was that we did not just cover what their assessment data meant. Instead, we thought critically about the valid uses and limitations of the data, and we worked through real-life problems where data could be used to inform instruction.
Another highlight was the work I did developing a credential based on a portfolio assessment. This was particularly challenging because the portfolio products took many different forms, even though they all reflected the same domain. I got to work with some very creative psychometricians and subject matter experts to research and implement an assessment process that was standardized enough to be defensible, yet flexible enough to accommodate a wide range of candidate profiles. The project was especially fun for me because of the unique measurement challenges inherent in a portfolio assessment.
What attracted you to the position at Questionmark?
Questionmark stood out for me because they are a company that takes reporting and analytics seriously. Some companies seem to treat assessment reporting as an afterthought–a byproduct of the assessment. But in many ways, the reports are the manifestation of the goals of assessment. We collect data so that we can make valid, informed decisions, and this can only be done with efficient, comprehensive reporting strategies. When I saw that Questionmark had a position dedicated solely to reporting and analytics, I knew that this was a company I wanted to work for.
Psychometrics: that’s quite a word! Can you describe what it means in terms that all of us will understand?
It is quite a word–one that spell checker can’t even handle. The most concise and comprehensive description I have read was from Mislevy et al. in the Journal of Educational Data Mining, 4(1). Psychometrics measures educational and psychological constructs with a probabilistic model. In short, psychometricians must take observable evidence (e.g., answers on a test) and make a probabilistic inference about something unobservable (e.g., the test taker’s true ability).
What do you hope to achieve as Product Owner of Analytics?
I am going to continue to build on Questionmark’s reports so that they best suit the needs of our clients and the changing practices in the assessment industry. I plan on using my background in psychometrics and reporting to make sure that our reports are designed and organized in a way that lets users quickly and intuitively leverage their data to make good decisions about their tests and testing programs. I am very fortunate because Questionmark already has a great reporting and analytics structure, and I am excited to continue that work.
How has the field of measurement and evaluation changed in the last 10 years?
In my opinion, measurement and evaluation has changed substantially over the past ten years as a result of improved technology. Fantastic work is being done to improve the accuracy, validity, and reliability of assessment. It is great to see organizations researching new item types, methods for shorter tests, and models that provide diagnostic feedback to test takers. Unfortunately, with such easy access to testing software, there are the occasional cases of “testing for testing’s sake,” where tests are administered without consideration for design or how the results will be used.
Overall, I am optimistic about the course of measurement. There is increased accountability, more support for methods like Evidence Centered Design, and better training and tools available for professional test developers.
How do you see psychometrics contributing to the future of learning?
Psychometrics is contributing to the future of learning by helping educators and students make sense of patterns in the data. While statistical models will never be a replacement for the expertise and instincts of a trained educator, assessments can be a handy tool for understanding students’ strengths, weaknesses, and work strategies. Tests no longer have to be about classifying who passed and who failed. They can now help provide diagnostic feedback so that educators and students can interpret their performance and adjust their learning accordingly. In this way, I think psychometrics has a large role to play in bridging the gap between grades and learning.
What are some of your current research interests?
Right now, I am interested in evidence models for task assessments like games and simulations. Technology lets us create some stunning virtual environments, but the research around how to score these assessments is still in an inchoate stage. While expert ratings and rubrics remain the standard for scoring these assessments, I am researching how we can model difficulty and discrimination in a complex task environment, especially as they relate to the student’s observed process for solving the task.
Tell me some things that interest you outside of work.
I love being around good friends, good food, and good music. I love to travel and go camping with my family, but most weekends you will find us puttering around our plot in our community garden or riding our bikes around the city. I also like working with my hands a lot–I help plant trees with a local nonprofit, and I try to fix stuff around the house (with mixed results).
If you’re attending the Questionmark Users Conference in Baltimore next week, please introduce yourself to Austin!
Posted by Jim Farrell