Skip to content

Ep. 36 – Assessment Luminary, Maria Hamdani, Center for Measurement Justice

24 Apr 2023
Share:

Maria Hamdani, Vice President of Assessment and Research at the Center for Measurement Justice, joins John Kleeman to discuss the role of assessments in promoting equity and justice in education and employment. She shares her insights on how assessments can be designed and administered in a way that is fair and unbiased, and how they can be used to identify and address systemic barriers to educational and professional success.

Maria highlights the importance of including diverse perspectives and voices in the development of assessments, and the need to continuously review and update them to ensure they remain relevant and effective. She also discusses the challenges of balancing the need for standardization and consistency in assessments with the need to accommodate individual differences and contextual factors.

Maria emphasizes the critical role that assessments can play in advancing social justice and equity, particularly in the context of medical education and licensure, and encourages educators and employers to embrace a holistic and collaborative approach to assessment design and implementation, one that is grounded in principles of fairness, transparency, and inclusivity.

Full Transcript

John Keeman:

Hello, everyone and welcome to Unlocking the Potential of Assessments, the show that delves into creating, delivering, and reporting on fair and reliable assessments. In each episode, we chat with assessment luminaries, influencers, subject matter experts and customers to discover and examine the latest and best practice guidance for all things assessment.

I’m your host, John Kleeman, founder of Questionmark and EVP of Industry Relations and Business Development at Learnosity, the assessment technology company. Today I’m really pleased to welcome Maria Hamdani. She started her career as a mathematics teacher and moved into creating and managing assessments. She’s worked at the New York City of Department of Education at Caplan, was the director of assessment of the college board and director of Curriculum instruction assessment at Curriculum Associates.

She has a bachelor’s degree in mathematics at UTA and a master’s degree in learning analytics from Teachers College, Columbia University. And she has recently joined the Center for Measurement Justice where she’s vice president of Assessment and Research. Welcome, Maria, and it’s very good to have you on the program.

Maria Hamdani:

Thank you so much, John, and thank you for having me. It’s a real honor.

John Keeman:

The question I usually ask people is how did you get into assessment?

Maria Hamdani:

Quite by accident in that, so I was a high school mathematics teacher. I taught for 10 years and then I had three kids in four years and needed to do something different. I was able to… That’s when I actually started working for Sylvan where I was teaching from home. And then, naturally just moved into this space of assessment. This was 20 years ago almost now.

And so, assessment at the time in publishing companies, it wasn’t such a formal process. And creating them. And so, I grew with the world of educational assessment at publishing companies and have been doing it ever since. And the plan was always to go back to the classroom. But my roles and my work has grown in such a way and taken such a path that I did not anticipate. And so now, I am doing something quite different, but the plan is still to one day, one day return to the classroom.

John Keeman:

Really, you’d like to go back to teaching?

Maria Hamdani:

Absolutely. That was an extraordinary experience for those 10 years and I can’t wait to go back. I miss the kids and I’m sure now when I go back I will be a very different type of math teacher, but I love mathematics, I love kids, I love teaching, I love education. And I love what I’m doing now.

John Keeman:

Well, that’s really interesting though. I imagine you are seeking to make a big difference to a lot of kids in your role. I know you’ve worked at almost all the big ed tech companies from Caplan to Curriculum Associates to Imagine Learning or whatever. Tell us a little bit about how digital assessments have changed over the last 15, 20 years.

Maria Hamdani:

Yeah, they’ve changed a lot. When I first started creating digital assessments, I think it was probably 19 years ago now. We used to create them on Word Documents and then the Word Documents, what were served to the kids and they were digital assessment. And then slowly there were different sort of software available. And that has come a very long way as you know. I mean Questionmark was one that I used at Kaplan, and now Learnosity is one that I’ve used for many years now.

And the capabilities that these assessment software companies have created to have digital assessment in different ways and serve students in different ways. Because it is really important too that students are able to be given different ways of showing what they can do. And the digital assessment space has a very high potential to support that. That is actually one of the four things that define an anti-racist assessment as per Dr. Jennifer Randall in her paper that she wrote in 2021.

And that is one of them, to allow for multiple ways of knowing and understanding and performing the content. And so, digital assessment softwares can certainly support in that. Now how they can, what we want to do with them, what we envision the possibilities to be, those aren’t there yet, but they’re getting there. Evolution happens. I mean, look at ChatGPT. Who would’ve thunk it? And so, just as that has evolved and will continue to evolve, I think that educational technology will, especially assessment technology will continue to evolve as well.

John Keeman:

I’m sure that’s the case. Let’s look at this cultural assessment and anti-racist assessment because I know that’s a passion of yours. I was speaking to somebody recently who I don’t really agree with, but they were saying that the diversity really belongs more in learning than an assessment. And that the assessment’s job is to measure. I’m sure you’ve heard this, the statement by lots of people. Tell us why it isn’t the case or if it is the case.

Maria Hamdani:

Well, yes, an assessment’s job is to measure. The question is measure what? What are we trying to measure? Yes, we’re trying to measure the standard from, if it’s a math assessment, if we’re trying to test, do they know how to multiply exponents? Yes, we’re trying to measure that. No doubt.

Does that mean that it’s not an opportunity to measure something else? Does that mean that it’s not an opportunity for students to be seen and heard and feel valued and affirmed? Of course it should be. I am a person of color and I grew up in England and the US. I’m Indian, my parents are immigrants. And never, I mean that’s a very strong word, but I would say never for me, of course, I graduated many, many years ago, never had I ever been represented, me, my culture. And that is who I am had never been represented in the assessments that I took.

And what does that say? What message are we sending to these kids when we don’t represent them? And in America alone, kids of color make up 54% of our country. We’re no longer minorities. And the educational materials that they see, and certainly the assessments that they see do not reflect that statistic.

John Keeman:

I mean you could presumably have, if we look at maths assessment for example, you could just have a very dry question. What is this number plus this number? Or how do you multiply X and Y? Or whatever, which presumably is culturally neutral. But I think what you’re suggesting is that there are questions out there which make assumptions about people’s culture or background, which often don’t represent enough different cultures.

Maria Hamdani:

That’s right. When we talk about items with context, mathematics items with context, oftentimes what we think is culturally neutral, there’s no such thing as cultural neutrality. There’s no such thing. There is a culture, period. Which culture are we presenting to the kids? That’s the question. So far to this point, it has been predominantly the white dominant culture that has been represented.

And I mean there’s so many different ways. And this has started, this is not just right now, I would even say the word, it’s a movement. That’s how I see it anyway. But this, it’s not happened just now. Maybe 10 years ago, I would say, maybe 15 years ago when it was we should put culture in. How do we do it? It started out with very, very simple things like a name. But not too hard to pronounce, but a name. A name that would not trip people up.

And there was lots of discussions around these kinds of things, and they were very superficial. But it was a start, and you have to start somewhere. Well, now we want to go deeper and further. And now we’re talking about culturally responsive and anti-racist assessment. And culturally responsive really is just saying, let’s have an item about another culture.

I mean, if you have a math item that talks about John is learning the guitar and he learns five new songs a week. How many songs will he learn in three weeks? Something very simple like that. That’s what people would generally say. That’s a culturally neutral item. No, it’s not. It’s absolutely not.

John Keeman:

Because John is not a culturally neutral name. It implies that he’s more likely to be white is I think what you’re saying. And the guitar maybe also is more likely to be done by richer people than less well-off people or some cultures than other cultures.

Maria Hamdani:

Sure, sure, yep. All of those things. I mean it would be a very simple substitution for that. Very simple. Just one example would be Raj is learning how to play the sitar. And he learns five songs a week. And how many songs will he learn through? All of a sudden, every little Indian boy or anyone that’s Indian or looks at that item and sees themselves, they are represented.

And that gets to happen. And when that happens, no other culture is deprived of anything. That’s one argument some people try to make. We’re eradicating whiteness. Nobody’s eradicating anything. It’s not an “either-or” situation. It’s a “both-and”.

John Keeman:

And so, a lot of the people listening to this podcast may well be people writing questions, some of them in K-12 or schools, others in other sectors. What advice would you give people who are either writing questions or involved in item writing teams or setting up item writing teams?

Maria Hamdani:

That is a great question. And I would say that advice that I would give would be to take a step back and first evaluate your process. What is the process in which you are creating these assessments? What is the environment in which you are creating these assessments? Who sits at that table?

Lots… It’s not just about the… The actual item writing part of it is definitely not the beginning. There should be conversations around what is it that we want to change? How far do we want to go? What can we do? Because publishing companies sometimes have their hands tied if they have to abide by rules of certain states. And we all know which states we’re talking about that say, “Absolutely not. No way. No how. And if you do, we’ll pull all of your stuff.”

There is something to be considered. There are considerations, but does that mean you can’t do anything? Of course not. Find a way to fulfil the objective, the vision that you set out for yourselves. And that’s number one, find your vision.

John Keeman:

I mean, if your vision, as I imagine most programs should be, is that you want to be completely culturally neutral and you want to promote diversity and inclusivity. How can you set up an item writing process that does that? Is it about ensuring that your authors and reviewers themselves are diverse? Or is it about item writing guidelines or what are good practices that work?

Maria Hamdani:

Yes, to all of the above, and but to some of that. And that is first of all, the term culturally neutral is not a thing. There’s no such thing as cultural neutrality. Then it’s like what are the cultures that you want to represent?

And yes, these do come into play in writing your guidelines, creating your training, having now should they all be people of color? No, the responsibility and the burden should not lie solely on people of color. That is a very difficult thing. Now, that does not mean that you should not have some writers and reviewers that are. But we get to also include everybody in this process that wants to be included, regardless of your culture or your race.

Once you create the assessments, then there are other things that you can put into place before and after. And those are things like having an advisory board. Having an advisory board of what you want represented. After they are created, having people that do content review and validate the culture that you’re trying to represent to make sure that you have authentic reviews of your content.

Because I actually wrote an item for something recently and it was about hair braids in the African culture. And I remember thinking, I knew about this and why wouldn’t I? And then researching it and thinking, “Oh my goodness, I actually know nothing about this.” And then, when my natural instinct was to relate it to a similar thing in the Indian culture. We have a thing called a paranda, which is basically a rope that you braid into your hair. And it was like the intersectionality here is fantastic.

And it’s things like this, it’s content like this that is interesting, it’s engaging. Kids see themselves when they see something like that. It’s exciting. I can’t tell you how many times… Well, I can tell you because there’s not too many. But when my kids would come home, “Oh, mama today they talked about this in school and it was really cool.” And it was just something so simple, so small, but they felt like they were important. And it’s important as educators that we do that.

John Keeman:

What’s the damage being done by assessments being culturally biased or racially biased? I mean, presumably people are not creating life opportunities that they ought to be getting.

Maria Hamdani:

That’s right. And it goes beyond that as well. It’s about where did this all come from to begin with? It’s a systemic issue. Assessments, from the beginning of time when they were created, they weren’t created with culture in mind. And there’s no purposeful bias against we’re going to do this so we can eradicate all of these different culture. That’s not happening. That’s not what’s happened here.

What has happened is systemic oppression has happened and it’s continued. And now the awareness of the fact that that has happened, it is still in effect in so many different realms. Assessment is just one of them. That’s the lane that I’ve chosen. But that doesn’t mean this is the only lane that it’s in, as we all know. And so, I’m sorry, I forgot your question already. What was it?

John Keeman:

No, it’s all right. Well, so I think it was the consequences of this. But I think also a follow-up is how much of this is a US issue and how much of this isn’t an international issue? Because obviously, there has been a lot of past oppression in the US and probably more subdued oppression in other countries as well. Or most countries in the world now are seeking to be very inclusive and racially equal in things. How much of an issue do you think this is in other countries in Europe or India or other places?

Maria Hamdani:

I think it’s everywhere. I think it’s everywhere. It may look different in different places. I taught in India. I taught in India for a year as well. And it was very interesting because here I am, Indian. And I thought I would fit right in because I’m Indian. But actually, my culture is quite different because I did grow up in the west. And although I know and love the Indian culture, it didn’t take very long before I realized, “Oh, wow, I am different here too.”

And that speaks to funds of knowledge, which is each child’s individual culture and what that looks like. For me, so and a lot of the kids that we have here in the US are like that. Are like I was, are like my kids are. And so, it’s an adaptive culture that we also get to represent. Now, is this an issue all over the world? Of course it is. Of course, I haven’t lived in every single country and I’m not an expert in every single country. I’ve lived in three, but people are people all over the world.

And I actually have a cousin of mine that’s doing this work in England as a matter of fact. And the same she says we face the same issues. And the one thing I did want to answer though, when you asked me what are the repercussions of not doing this, it is that kids, they feel… When we make kids invisible, when we don’t value them and they feel invisible, they grow up to be adults that are not valued and feel invisible.

And we are all going to the same workforce. And these issues that happen in schools and the confidence levels that we are really crushing for these kids, that stays with them as they become adults. And it’s a horrible, vicious cycle. And it just gets to stop. It just gets to stop.

John Keeman:

I mean, we are recording this in late March. And I’ve just come back from the ATP conference in Dallas. And there’s a lot of… I mean the assessments, psychometricians and other experts, they’re very, very keen to make their assessments fair and valid and reliable and inclusive.

And there’s a DEI committee of the ATP and lots of people working in this space. We talked a little bit about some best practices before, have an advisory board, have item writing guidelines, do a lot of review. I mean, I think a lot of people passionately want to do the right thing. Can you suggest any more practical measures that people can do using technology or not using technology to make better assessments that are fairer?

Maria Hamdani:

Yeah, that’s a good question. I think, again, it’s a matter of being mindful about it. It’s a matter of finding the right partnerships for those people that are doing this research. And it will take research, it will take creating assessments that are different and they serve a different purpose. I shouldn’t say a different purpose. They serve an additional purpose.

And that is not just to test the assessments, but to also make students feel safe, validated, affirmed, seen. And how do we do that and how do we leverage technology to support us in doing that? And there are a few different ways, but what I would say, if you’re a publishing company out there and you’re thinking, “How do I start this work? How do I go about doing this? How do we make sure we do it right?”

Find the right partnerships, find the right people that are doing the research in this field. And let them continue to do the research to support the foundation of what it is that you are trying to create.

John Keeman:

And in terms of digital, what can digital vendors like Questionmark or Learnosity or other companies, what should we be doing to help support people who want to make fairer assessments?

Maria Hamdani:

That’s also a great question. And I would say a similar response in that partner with the right people. I mean center of measurement justice, we are partnering with Learnosity as well to figure this one out. But different things. I know that I believe there’s… I don’t know which company, but somebody’s already trying to do foreign language support for students that speak as a different language.

There used to be a time where, if you spoke a different language, it was considered a negative. I know that when my own kids started school, none of them spoke English till… Well, the third one did because she was the third one. But the first two did not speak English till they went to school. And when they went to school, you have to fill out a form that says for their language, what is their first language.

And I remember telling my friends, “Do not put their first language as Hindi or Gujarati or whatever it is from home. You put English.” Because if you do, they will be placed in classes for the entire elementary school all the way through sixth grade, they’ll be placed in certain classes that they don’t need. They will have a tag on them and it will be a negative one.

They’re all watching TV. They’re all watching Dora the Explorer and all those little shows that they watched when they were little. They’ll manage just fine. Within months, me and their father constantly, “Nope, don’t speak English in the house. Don’t speak it.” Because then, of course. And now they can speak their language and they can understand our home language. But of course English is it. Of course, English is it.

I think foreign language support and really just understanding that it’s not a negative. Even things like if you’re creating a passage in ELA. Let’s leverage technology to where, in that passage, they get to choose the pronouns of that story, he, she or they. They get to choose. Things like pop-up glossaries or even pop-up author bios in ELA so we can show the diversity that we are using diverse authors.

Then there are something that we’re actually trying to… We are researching at CMJ. And that is scenario-based assessments in mathematics, which is similar to what we do in ELA. We have a passage and then you have items.

John Keeman:

I’m sorry, ELA is?

Maria Hamdani:

Oh, I’m sorry, English language arts, reading.

John Keeman:

Right, right. Yep, yep.

Maria Hamdani:

Sorry. And in mathematics, right now it’s, here is a question, answer the question. Well, in creating anti-racist assessment, we have realized that you can’t do it without a certain amount of context. Especially right now, when people have not been exposed to anti-racist assessment.

And so, there is some explaining to do, sometimes within the item. That goes against the rules of educational psychological testing, that big book with all of the rules. It goes against that because, according to those rules, it is construct irrelevant. In other words, you’re putting in things into this item that are not needed to solve the math problem.

That is true. We are also trying to solve a different problem and that is why. And so, how can we justify putting so much text in? Well, how about we do scenario-based mathematics? You have a story. And in the story that is engaging, the math questions are built into it. And for this, we would definitely need the use of technology. What kind of technology? Perhaps there’s a cartoon that plays while you doing it. It’s all mathematical. Or maybe it’s storyboards. Or whatever that’s going to look like, we’ll need the support of the assessment development companies, these Learnosity and so forth.

John Keeman:

Oh, indeed. It is actually our company that has produced, or there may be other companies as well that produced something for instant translation of questions. You can have the question up on the screen and then click on some of the wording. And you can instantly translate it to different language to help people who are non-native speakers.

Because I think there’s something like over 20% of people in the US speak a different language than English at home. With a lot of that Hispanic, but other languages too. What about? I think many people listening to this podcast may be working in different kinds of assessment, IT certification or recruitment exams or corporate development exams. Do you think these ideas apply there as well?

Maria Hamdani:

Absolutely. Absolutely. They apply.

John Keeman:

And what? And then it’s just a question of making sure the context includes things that are applied to different cultures.

Maria Hamdani:

That’s right. Different cultures and those are the culturally responsive ones. And then to make them anti-racist, you want to anti-racist assessments, again, according to Dr. Randall, explicitly disrupt conventional negative stereotypes as they relate to many marginalized groups. And then it highlights these oppressive sociopolitical inequities and injustices. And hopefully if it’s done right, will empower students to enact change. This kind of empowerment is not only a positive but is needed in every realm, be it K through 12, higher education or the workforce.

John Keeman:

How fast do you think things are going to change?

Maria Hamdani:

Oh, I think it will take at least a generation. Yeah. I mean if I get to see the change in my lifetime, I’ll be happy.

John Keeman:

And what does the change look like? That really assessments are genuinely culturally neutral. Even though you said it was difficult to achieve, but that there is no divergence between different cultures, how they approach assessments and get results from assessments.

Maria Hamdani:

I think for assessments to be anti-racist is I think what… I mean, there’s culturally responsive and then there’s anti-racist. And culturally responsive is definitely… Or culturally sustaining, which means it really talks about and supports the culture of the area, the community. That is what’s really important as well. And I’m sorry, can you repeat your question? I think I lost the question.

John Keeman:

I think it’s how do you see the, when things are better, what will they be essentially?

Maria Hamdani:

Yes. Well, they’ll be, in a word, an anti-racist society.

John Keeman:

Fair enough, fair enough. New society to change to. And okay, just as we close this podcast, some calls to action. Most people listening to this, I imagine they’ll want to do the right thing. Three things that they could do and go out to change tomorrow or improve tomorrow that would help get us on the way.

Maria Hamdani:

Number one, understand your own biases. Self-reflect, understand what those are. Because you can’t do this work until you can know who you are, understand that and accept that and then educate yourself. Number two, partner with the right people, organizations, teams, researchers, scholars, whatever, whoever that makes sense for the context in which you do your work. And number three, create a vision and an intention, and then execute. Intention is 90% of this, 10% execution, as is with anything in life.

John Keeman:

Thank you, Maria. That’s so great to hear. And I hope that might be actionable for some of our listeners. And thank you, all our listeners, for listening to us today. We appreciate your support. And don’t forget, if you’ve enjoyed this podcast, why not follow through your favorite listening platform? Also, please reach out to me directly at john@questionmark.com with any questions.

And I’m sure also you can reach out to Maria at the Center for Measurement Justice. And they’ve got a great website with a lot of resources about this. Or you can also visit the Questionmark website at questionmark.com to register for our best practice webinars. And thank you for listening and please tune in again for another exciting podcast discussion we’ll be releasing shortly.

Related resources

Get in touch

Talk to the team to start making assessments a seamless part of your learning experience.