Join host John Kleeman in an insightful conversation with Dr. Ada Woo, VP of Innovative Learning Sciences at Ascend Learning and seasoned expert in assessment and psychometrics. Dr. Woo shares her journey from aspiring clinical psychologist to becoming a leading figure in the assessment industry. They discuss various topics like the role of psychometrics, the importance of fair and reliable assessments, the impact of test anxiety, and the evolution of fairness in testing. Dr. Woo also delves into the integration of AI in assessment and learning, emphasizing the need for adaptability and innovation in the field. Tune in for a deep dive into the complexities and advancements in educational and professional assessments.
Full Transcript
John Kleeman:
Hello, everyone, and welcome to Unlocking the Potential of Assessments, the show that delves into creating, delivering, and reporting on fair and reliable assessments. In each episode, we chat with assessment luminaries, influencers, subject matter experts, and customers to discover and examine the latest in best practice guidance for all things assessment. I’m your host, John Kleeman, Founder of Questionmark, and EVP at Learnosity, the assessment technology company.
Today, really pleased to welcome Dr. Ada Woo. Ada has a Ph.D. in quantitative psychology from the University of Texas at Arlington and has nearly two decades experience in the assessment industry. Ada is Vice President of Innovative Learning Sciences at Ascend Learning, where she oversees the R&D of learning and assessment content. Prior to joining Ascend, she had senior roles at ACT and at the National Council of State Boards of Nursing, and she’s also been a volunteer leader at the NCME and the ATP, the Association of Test Publishers, and she currently serves with me and others as a director of the ATP.
Hi, Ada. Very pleased to have you here.
Dr. Ada Woo:
Hello, John. How are you?
John Kleeman:
I’m very good. I’m very good. Recording when it’s very cold outside in England. What’s the weather like for you?
Dr. Ada Woo:
Oh, it’s pretty cold in Iowa too, in the States.
John Kleeman:
I bet Iowa is even colder than here. What I tend to do is start by asking people how they got into assessment, so how did you get into assessment?
Dr. Ada Woo:
Yeah. So ever since I was pretty young, I had aspirations to become a clinical psychologist. So I went to university, I majored in psychology, and as I learned more and more about clinical psychology and I did an internship at the mental health facility, I realized that path is really not for me. My passion is somewhere else. And as I progressed in my graduate school career in my master’s degree, I did my thesis on test anxiety. And as my thesis project, I started to look for assessment and screening tools for test anxiety and I saw some that I didn’t think were very good. So I started looking into how those assessments were made and what is the science behind it. And one thing led to another, I found my passion, and that’s where I ended up as a psychometrician.
John Kleeman:
So it’s a pretty basic question, but what is a psychometrician? I think our audience might like to understand your perspective on that.
Dr. Ada Woo:
Yeah, psychometrics is basically the science of assessment. So psycho comes from psyche, the Greek word for the soul of the mind, so measurement of the mind and measurement of something generally you can’t directly see. And in psychometric speak, we call that a hidden construct, a latent construct. So that is a science that we develop methodologies for us to be able to get to a latent construct, something that you can’t readily see, such as knowledge, skills, ability, whether someone will be a good nurse or a good physician, et cetera, or whether someone is depressed or anxious during a test. So those are all the things that a psychometrician might be interested in looking at.
John Kleeman:
And I know a lot of psychometricians, including yourself, have doctorates, but is it a requirement to have a PhD?
Dr. Ada Woo:
Personally, I don’t think so. I think generally, you would need a graduate degree. I think for a variety of positions, the master’s level will be sufficient, and generally, it’s a graduate-level training kind of career, but I don’t think a doctoral degree is completely necessary for all positions.
John Kleeman:
Thank you, thank you. And I’d like to come back to the test anxiety point as well later, but let’s just run through a little bit about what you’ve done in your career. What was your first job and how did you move on after that?
Dr. Ada Woo:
After I finished my graduate training at the University of Texas at Arlington, I moved to the Washington, D.C. area to be an operational psychometrician for the physical therapy exam, the physical therapist licensure exam in the United States. And I learned a lot there, and after a couple of years, I saw an opportunity came up at the NCLEX exam, the National Council of State Boards of Nursing, and they are the provider of the licensure exam for entry-level nurses in the United States and much of Canada, and I thought that was too good of an opportunity to pass up. So I moved to Chicago and I was very fortunate to have very good mentors at NCSBN, and then I was provided opportunities to take on people management duties and learn in that way. So my career kind of moved on from there.
In 2017, Alina von Davier, whom I’ve known for quite a while, she was the founder, a founding vice president of ACTNext. So at the time when Marten Roorda was CEO of ACT, he wanted to diversify ACT’s portfolio to expand beyond just the assessment, the ACT test, the paper and pencil. He wanted to expand into adaptive learning, ed tech component, more of a comprehensive learning and assessment company. So Alina was recruited to start ACTNext, and then Alina recruited me to work with her. So we did that for a few years and then an opportunity came up in Ascend Learning and here I am.
John Kleeman:
And so who are Ascend Learning and what do they do and what do you do with them?
Dr. Ada Woo:
Ascend Learning is an education provider, primarily in the healthcare market. We have three segments: healthcare, safety and security, and personal wellness and fitness. So we focus on high-growth careers and we focus on getting people into those careers, excelling in those careers, and being successful in those careers through our assessment and learning product. And my role is that I lead the Innovative Learning Sciences Division and my team is responsible for creating and developing all the assessment and learning content for our healthcare portfolio, which includes nursing, medicine, and health sciences.
John Kleeman:
So what’s a typical day for you? I mean, I know you and I have both got an ATP board meeting later this morning, but what are you going to do this afternoon or what are you going to do next week?
Dr. Ada Woo:
So today is actually a very special day in Ascend. We have a day of wellbeing, so we encourage work-life balance with our team members, and it’s been a challenging year. It’s very busy for everyone. So today is actually our day off, so the company is taking a day off.
So this is not a typical day, but on the typical day, I would say I spend a lot of my time helping team members prioritize because there’s, I’m sure the same for you, there’s always more to do than there’s time to do them all. So I help team members prioritize given the resources that we have, and also work with my cross-functional partners in technology and product to determine the priorities of our project, how we’re going to roll it out and in what way are we going to develop those products.
John Kleeman:
That’s interesting. And I guess a lot of managers end up helping their team prioritize as a key thing.
So look, let’s segue a bit. I’d love to talk about DE&I because I know that’s something you’re very knowledgeable about, I’d love to get back to test anxiety at some point, and a little bit about advice on good practice and assessments. Should we start with DEI? Why is that important in assessment and what’s your view on it?
Dr. Ada Woo:
Yeah. I think assessment, as an industry, we serve a pretty vital function in society, both in the education and professional space. I see us as having the role of ensuring public safety, making sure that, for example, people who are physicians, they know what they’re doing before they get their license. But on the flip side, we’re also providing the function of providing access to people who may not normally have access to education opportunities and professional opportunities, for example.
So at my former employer, the ACT, many students would take the ACT as a means to gain entrance to universities, and many first-generation students will take the ACT for scholarship opportunities, for example, the opportunities that they might not have otherwise if they don’t have a standardized test to help them demonstrate what they know.
To me, it’s a personal story. I am a first-generation college student, so I benefited from doing well on standardized assessment and those assessments opened doors for me to get into universities, to get financial assistance, to get into graduate programs, et cetera. So I’m a believer of what we do.
John Kleeman:
So a lot of people listening to this podcast will be assessment practitioners. I’m sure a lot of people, almost everybody wants to be more inclusive and more diverse, what sort of recommendations or advice would you give them?
Dr. Ada Woo:
Yeah. I mean, I think there are a lot of good resources and best practices guidelines that is in the profession that I think that will be a good place to start. Of course, the Standards for Educational and Psychological Testing is a good place to start, and there’s quite a bit of information on how to make a good test, how to make a fair, reliable test, including a DEI aspect. And of course, our newest publication in collaboration with ITC, the ATP/ITC Guidelines for Technology-Based Assessment. I think there’s quite a bit on there as well on how to make a fair and reliable assessment, technology-based assessment that is inclusive.
So I think first off, I would encourage folks to read the relevant chapters and match it up to their current process and see if there’s any gaps or any additional things that we can all do. And I think we just need to acknowledge and recognize that we all come with our own biases and our biases influence how we work. And I think that will be the very first step, right? Whether you are a member of an underrepresented group or a disadvantaged group or not, I think everyone has their biases and understanding that will help us make better decisions as we create our assessment and learning products.
John Kleeman:
And you mentioned at the beginning that you’d done some work on test anxiety. Is that also a diversity issue? And what can we do as test developers to reduce test anxiety?
Dr. Ada Woo:
This is not even a DE&I issue; it’s just about making a good test. So, before I became a psychometrician, like many people, I thought a test is very easy to make. It was just two people sitting in the room writing a test and the purpose of the test is trying to, you know- the test maker is trying to trick you into answering it wrong, so have tricky questions. And now being a psychometrician, I know that is a very bad way to make a test, and most people don’t do it that way. But I think just making the test based on best practice guidelines, don’t ask trick questions, be clear in your statement and things like that, I think that will alleviate quite a bit of that test anxiety, making the test questions easy to read and at a readability level that is appropriate.
And in terms of test anxiety, I think a lot can be done in accommodation, whether it is if it’s appropriate, giving someone extra time or a separate room, a separate location to test rather than sitting in the main hall with a bunch of other students. So if it is appropriate, exploring the way that the test is administered will also help.
John Kleeman:
And do you think people should need a diagnosis to get that or it should be reasonable that people ought to be able to just ask for it and say that they’ve got test anxiety?
Dr. Ada Woo:
I think for high-stakes assessments in terms of fairness, we probably need some kind of diagnosis or some kind of professional assessment. Not just because I’m stingy with accommodation, “I don’t want to give everyone accommodation.” Accommodation is meant to be helpful to someone that needs it and there are so many different types of accommodation. And I think having a professional assessment or professional help in saying that, “What is the issue of this particular test taker,” also helps the test provider to provide the best accommodation that they can use and will help them demonstrate their skills.
John Kleeman:
And how common do you think test anxiety is?
Dr. Ada Woo:
I don’t know the figure, so I haven’t read up on the statistics, but I think based on my work, most of what I do is to work with education institutions. For example, our biggest brand, ATI Nursing Education brand, we provide assessment and learning content to nursing programs and universities and we see quite a bit of accommodation requests for test anxiety for folks that need maybe extra time in separate room. And we will work with the education providers to help with that. So I think it’s probably more common than we think, but I don’t know the specific statistics.
John Kleeman:
No, fair enough. And you mentioned fairness. Do you think fairness and diversity are almost the same thing or a little bit different?
Dr. Ada Woo:
I think people usually use the two words almost interchangeably. One point I like to make is this: I think the perception of fairness is changing in the assessment and the psychometric realm. When I was learning about assessment several decades ago when I first started learning about how to make a test, fairness means that everybody gets the same test, everyone gets very, very sterile test questions. You only include what is absolutely necessary to test the knowledge or skills in your questions. So you don’t add any context that’s irrelevant. Everything is super sterile and everybody gets the same.
And now, I think more research is coming out, especially in education assessment in, for example, formative assessment, that it is important to add context so students and learners can see themselves and they can feel like they’re part of this education curriculum. You’re talking about them, it’s something that they feel a sense of belonging. So I think now people are starting to add some of that “irrelevant” context back. And so I think the science is evolving and I’m glad to see that more emphasis is being put on inclusion and diversity.
John Kleeman:
That’s really interesting. Thank you, thank you.
So another big topic in the assessment community is how much you should be testing on recall and how much on other kinds of skills. In terms of the assessments you’re involved in, do they have a lot of recall and what are your thoughts about recall or other ways of testing people?
Dr. Ada Woo:
I think depending on the level of learner, so if you are in an elementary school program and you’re teaching young kids how to spell, I think there’s a role in recall and rote memorization. But if you are doing the type of assessment that we do in Ascend, we primarily try to get people into a practice-ready state, whether it’s as our personal fitness trainer or our nursing student or our phlebotomist, or physicians, we want to get these students and learner to the state that they’re practice-ready. And in those professions, being practice-ready, I believe, is not just a matter of recall, that you might need to be able to apply, to think critically.
And I also think that the need for recall might be changing now. I really can’t remember anything personally. Everything that I want to know, I can Google, I have my phone, I have my digital references. So I think that the need to remember all the facts is also changing with the digitization of things and the use of technology and AI, for example. The uniquely human things that we can do, the critical thinking, the application, those are things that we need to be testing on.
John Kleeman:
If other people are trying to write tests above recall, because I know that some people do find it a lot easier to write recall questions, what advice would you give them about how to author or review or create questions that test above recall?
Dr. Ada Woo:
Yeah. I mean, I think one of the very common tool that a test developer would use is grade every question according to the Bloom’s Taxonomy, or write according to the Bloom’s Taxonomy guidelines: recall, knowledge, application, et cetera. So I think that will be a good place to start.
I mean, really application questions are much harder to write than recall questions. So setting up your item writing, content writing, and content review process differently will help get those items to a higher cognitive level. Training your subject matter experts to think that way will also help. And another thing that might help is instead of writing just a fact-based question, maybe look at a case scenario.
So one of the things that we’re doing in nursing education in the US and Canada, in the nursing licensure exam earlier this year, NCSBS started rolling out their clinical judgment item type. So instead of fact-based items that might be a multiple choice, some of the questions might be a case scenario or an unfolding case study that you’re presented with a patient and some concerns, and then there are a few questions that travel along with you to see how the patient is progressing, and the student will have to make decisions on how to intervene and treat the patient. So those are clearly much more difficult items to write, much more costly to prepare, but I think it will help you probably get more out of it in terms of critical thinking.
John Kleeman:
And can you talk a little bit more about them? So are they multistep, and the second question will depend on your answer to the first question, that kind of thing?
Dr. Ada Woo:
They’re independent, but they build upon the NCSBN Clinical Judgement Models. So they develop a judgment model. I shouldn’t say, “They.” I was one of the original authors of the judgment model, so we developed the judgment model and they kind of follow the nurse’s clinical judgment and decision-making journey. So you walk into a room of a patient, you recognize a cue, you process the cue, you figure out what to do, and then there’s a feedback loop to say, “If your treatment is successful, what are you seeing, et cetera.” And those clinical judgment items in case scenario, they almost use the Clinical Judgment Model as a rubric. And those different elements in the Clinical Judgment Model will be included in each of the cases to make sure it encompasses the different elements that’s necessary for nursing clinical judgment.
John Kleeman:
And then people then will choose each step, the judgment that they need to make to diagnose or progress or triage or?
Dr. Ada Woo:
Yes.
John Kleeman:
And do you think those would also be applicable in other sectors, that kind of concept?
Dr. Ada Woo:
Absolutely. I mean, in healthcare, I can see a lot of applications. I think we have in a long time, the medical assessment side of things, they use standardized patients. So I think it is the same concept. However, a standardized patient is much more costly and difficult to scale. So I think this clinical judgment model is combined with computer-based testing, computer-based delivery to make it much more easy to scale, to be able to apply that to a large number of examinees, to be able to grade immediately, and to be able to provide results faster. So I think that’s the thing that it can be applicable to many professions.
John Kleeman:
Just to paint a little bit of a picture to people about how these questions work, do they have a picture of the patient or a video of the patient or how do they set the context and what kind of answer do people give? Is it multiple-choice or something else?
Dr. Ada Woo:
Some of them, it’s multiple choice, some of it will be ordering a procedure, so they are a selected response item. So there are no essays or anything constructed response like that. Some of it might be highlighting out of a case note, a patient note, the relevant information to see if a nursing student knows what are the cues that are important for this patient. So a lot of it is, in a way, mimicking what a nurse would have to do at the bedside, reading a patient’s note, looking at a patient’s case file, deciding if a certain symptom is something that is important or a certain cue is meaning that this person is getting better or the medicine really doesn’t work. So those are some of the components in the scenarios.
John Kleeman:
Thank you. Thank you. So there might be some patient notes up there that they have to interpret or something like that?
Dr. Ada Woo:
Mm-hmm.
John Kleeman:
Okay. So, Ada, we ought to talk about AI. What are your thoughts about how AI is going to change assessment and learning?
Dr. Ada Woo:
I think AI is going to be transformative with assessment and learning. Just in the very recent example of Chat GPT, I think that is the fastest, most widely adopted product that’s ever launched. So I think that is very telling, that folks are finding ways to use it. And I think in the education realm, that’s definitely something that we want to explore.
In terms of just more general in the AI space, I see a lot of promise with adaptive learning. I’m sure you have very, very good teachers in your past that you feel like when you need help, you get a little more help from the teachers, or when you need to be left alone to learn in your own pace, he or she will let you do it. And I think having AI and technology to mimic some of that magic of a very good teacher, to be able to tailor the learning and the curriculum and the feedback to individual students, and I think that is something that AI can shine there, to combine learning and assessment and tailor it to the specific learner.
John Kleeman:
And potentially to bring the best teaching to everybody?
Dr. Ada Woo:
Yeah, I would say bring best teaching to more people because if you have unlimited time and resources and you have really good teachers, there’s not really a whole lot that you would need in terms of technology if you have access to that level of teaching and remediation. And we all know that access to opportunities are different or vary, and using technology, I think we can bring some of that magic to more children, more learners.
John Kleeman:
And lastly on AI, what about DE&I and AI? Do you think that AI will increase diversity and inclusivity, or reduce it?
Dr. Ada Woo:
I think AI has the potential to increase diversity, but the tool is only as good as what we teach it to do. So I don’t think AI in itself is the be-all and end-all magic solution. I think AI is just a tool and it’s only as good as the training data that we feed into the learning model. And so if we’re not careful, AI has the potential of just perpetuating the biases that we’re seeing in society today. So I think we need to be more cognizant of maybe even over-representing some groups or things that we want to increase the diversity for and to train the model to do that instead of just copying what we’re seeing today in society already.
John Kleeman:
That makes sense. So is there anything else you’d like to share? Any advice you’d like to give to people who are starting their assessment journey? What should they do to make good assessments?
Dr. Ada Woo:
In terms of a new assessment program or a new assessment professional?
John Kleeman:
It could be either or both. Whatever you like.
Dr. Ada Woo:
Okay. I think for a new assessment program, I’m a psychometrician, so I see the value of psychometrics. Whether you hire your own psychometrician in-house or contract with a reputable firm, that can give you good advice. If it’s a brand-new program, I would definitely recommend going with knowledgeable psychometricians to help you design the program, combined with subject matter experts, and kind of guide through that process. I think a good design from day one will save a lot of heartache down the road. Less to fix. So that’s the new program side.
I think for new professionals, I think many people get into our profession through the technical route, whether you are a psychometrician, a data scientist, an engineer, a developer, or learning designer, many people get into the assessment business because of their technical expertise and because of their training. I would encourage folks to also develop skills for communication and skills to be able to share ideas and collaboration working with different folks. So some of the 21st-century skills that we often talk about, I think looking at those and developing those along with the technical skills would be quite important.
John Kleeman:
Thank you. That’s really great advice and thank you very much for being with us today, Ada. Thank you to our listeners as well. We appreciate your support. If you’ve enjoyed this podcast, why not follow us through your favorite listening platform? Also, please reach out to me directly at john@learnosity.com with any questions, comments, or if you’d like to keep the conversation going. You can also visit the Questionmark website at www.questionmark.com.
Thanks again. Thank you, Ada, and please tune in for another exciting podcast discussion we’ll be releasing shortly.