Skip to content

Ep. 35 – Assessment Luminary, Norihisa Wada, EduLab

29 Mar 2023
Share:

John Kleeman is joined by assessment luminary, Norihisa Wada, executive adviser of EduLab in Japan, and director of the Japan Institute for Educational Measurement. Nori has experience with AI-driven voice recognition and has worked with Carnegie Mellon University and Kyoto University. He is also involved in EdTech investment. Delve into his insights on assessment and his background in the industry. Nori shares how he transitioned from his work on game development for consumer educational games and fitness games for Nintendo to the assessment industry, the future of assessments and gives advice on how assessment organizations can be more inclusive.

Full Transcript

John Keeman:

Hello everyone, and welcome to Unlocking the Potential of Assessments, the show that delves into creating, delivering, and reporting on fair and reliable assessments. In each episode, we chat with assessment luminaries, influencers, subject matter experts, and customers to discover and examine the latest and best practice guidelines for all things assessment. I’m your host, John Kleeman, founder of Questionmark and EVP of Industry Relations and Business Development at Learnosity, the assessment technology company.

Today, I’m really pleased to welcome Norihisa Wada who is executive adviser of EduLab in Japan and also director of the Japan Institute for Educational Measurement, which is a large Japanese exam board owned by EduLab. Prior to joining EduLab, he was a senior executive at a company who produced many of the world-famous gaming titles for the Nintendo DS and Wii, including the brain fitness games. He’s got a lot of experience with AI-driven voice recognition, has worked with Carnegie Mellon University in the US and Kyoto University in Japan, and he is also actively and materially involved in EdTech investment. Nori, welcome to the podcast.

Norihisa Wada:

Thank you, John-san. Hi. It’s my honor to be here talking to you.

John Keeman:

Well, we’re very, very, very pleased to have you. Look, the question I always start with here is how did you get into assessment?

Norihisa Wada:

Okay. My business intersection with the assessment industry first began, I think, 15 years ago, back in 2007. At the time, I was executive vice president for a company which was referred to as Nintendo’s Education Lab. And I hope, John-san, you know Nintendo. It’s a famous gaming company.

John Keeman:

Of course. I’m sure all our listeners will have played with… Go ahead.

Norihisa Wada:

Great, great. And in this company, we have created numerous consumer educational game titles and fitness games for Nintendo DS, 3DS, and Wii platform. And most of our titles sold over multimillion copies worldwide. And as you may recall, this was way before the smartphone ages. Each of the copies cost probably average $49 a copy. We were very successful in the business. And just to ring a bell, brain fitness games for Nintendo 3DS was developed by my team. And during this time, I toured around the world. I was very fortunate. I had a chance to tour around the world as a keynote speaker and panelist on behalf of Nintendo in academic conferences, mostly conferences like Gamification and Serious Gaming.

And then I was approached by ETS to exchange ideas on gamified learning. At the time, my mission statement for what I was doing was fun to learn. So I remember us discussing fun to take tests and stealth assessment in extension to the terminology of fun to learn. And eventually we, meaning ETS and my previous company, came out with ETS official toy for Nintendo DS, which was released in the year 2009. And this was my first encounter with the assessment industry. It was an exciting experience for me to learn rigorous academic research behind psychometrics, designing assessments, and maintaining operational excellence. As a student in the past, I took many exams but I didn’t know that there was science behind it. And I was very much amused by the interdisciplinary, multidisciplinary gaming projects, meaning game mechanics, design, and learning and measurement science intersected. So basically through Nintendo DS project, I got to know the assessment industry.

John Keeman:

Very interesting. And I’m sure lots of us… I remember when my son had an Nintendo DS and I used to play with it quite often and we had a lot of fun and I’m sure some learning as well. What was your background before working for Nintendo’s game publisher?

Norihisa Wada:

Before working for Nintendo, I was a strategic and business development officer for a voice recognition company. This, as you know, we see voice recognition technology everywhere today. But this was in 1998 when I joined this voice recognition startup, and my mission at the time was to draught strategy as a strategic officer for this state-of-the-art new core tech, which we won two consecutive years in the US DARPA project and create market around it.

I first targeted entertainment games industry and education industry as a market penetration strategy. The reason for this was voice recognition was still new at the time, so I wanted to have many audiences try out our new technology through fun experience. And that’s a huge impactful way to approach consumers. And also, why education industry? Because there was a natural fit of using voice recognition technology in some of the English learning practices, like repetition types of practice and using voice recognition to measure that. And eventually, we were lucky that Toyota saw a potential and became one of the early deed investors, and I was able to make this company public in 2005. And that’s what I’ve been doing.

John Keeman:

When you moved from games development to assessment, what interested you in assessment? Why make the jump?

Norihisa Wada:

During the time I was working for voice recognition technology, I was satisfied because we were… Well, at least I was able to market the state-of-the-art technology. But after making the company public, I started thinking, “Yes, core tech, advanced technology, very, very exciting.” But it’s a part of the product. You do not own the end product, so you are basically setting the enabling technology. And I wanted to see and explore designing the end product and see the impact. That’s why I had a connection to the gaming company. And at that time, Nintendo had a code name for Nintendo DS development and they were thinking of creating a new console, which was the Nintendo DS, was double screen, stylus pen, which happened to be a good fit to educational games. And everybody know today that Nintendo DS made a magic in terms of interconnecting education and games.

John Keeman:

I think that you became very interested in, like you were saying, fun to learn and making it fun to assess or that people were being assessed without realizing they were being assessed. Do you want to talk a little bit to that or explain the stealth assessment thing?

Norihisa Wada:

Sure. During the years of Nintendo, developing educational games for Nintendo platform, since it became such a big trend, Ministry of Education one day came to us and said, “We would like to fund a three-year project where you will get to use a public school environment and see the impact of games in normal classroom environment. And I obviously had a chance to interview teachers, district board of education, parents, and students who participated in the project throughout Japan. And so we were surprised by the power of games, and I wanted to scientifically find out what the secret sauce was that hooked everyone, making them happy and concentrated in certain types of learning. And I found out that successful games was full of science.

So in this academic journey was when I started teaching at Kyoto University Medical Centre to post-graduate students. And what I was teaching was habit design. I had the both experimental environment which was funded by the government, the public schools, and the academic lab which Kyoto University provided to sort my thesis and findings. And like you mentioned, the stealth assessment, one of the findings I had was that one, measurement done right in the form of stealth, and two, the interactivity and the feedback design as means of intervention is very important in learning outcomes improvement. And I just want to touch where this connects to my experience of designing games is I have two simple examples. For example, I hope John-san, you have played Mario Kart before.

John Keeman:

Yes, I have. I have.

Norihisa Wada:

Okay, good. Great. Just for the sake of the audience, a simple explanation about Mario Kart is it’s basically a racing game and you get to play multiplayer as well as single player. And when you first run in a Mario Kart course, you run. And of course, your lap time is not excellent. There’s so many places you can improve because it’s your first time seeing the course. But Nintendo was creative that they created what they call a ghost.

Now, ghost comes out when you play single alone and second time. After the second time, there’s always a ghost of a previous course that you’ve driven. And this is a stealth way of measuring your ability, and you get to compete against your ghost. Basically, it gives a good scaffolding for you to improve on top of that. And another measurement, or should I say an assessment type of tactics, used in games is, example, card games. Let’s say if you’re playing full player card game, one is you and three is all run by computer, AI. In a card game, one of the player, AI player, will start imitating how you play as a card game hand. And you start realizing that this AI-ran player is similar to you. And this is a way of objectively showing you or making you realize the pros and cons of your tactic as well as your ability.

And all this is based on stealth measurement. The user, just the player is just having fun. They don’t know they’re being measured. But with a good feedback and very interactive intervention, it becomes a very, very powerful tool for an improvement of players and learners.

John Keeman:

Very interesting. Essentially, you are maybe playing as yourself and learning that way and making assessment like that. Tell me, Nori-san, and about your current role because I know you’ve now working for EduLab and also JIEM. Can you explain who those are and what you do there?

Norihisa Wada:

Sure. First of all, John-san, as you already know, I wear multiple hats. And so I am currently an executive advisor for EduLab, and I was formerly a chief marketing officer and chief operating officer for EduLab, as well as director of JIEM. And JIEM is 100% owned subsidiary of EduLab, and JIEM stands for Japan Institute of Educational Measurement. Basically, EduLab is the parent holding company. And I am also an executive advisor for one of the largest shareholders of EduLab, which is the Zoshinkai Holdings. And they are the most famous distance learning company in Japan with a prestigious history.

And JIEM, in a very simple way of describing their functionality, JIEM is basically like the ETS, ACT, Cambridge Assessment of Japan, and College Entrance Examination Board combined. JIEM does many national assessments funded by the Ministry of Education. All of the national assessments conducted in Japan is done by JIEM. Global assessments like PISA, PIAAC, TIMSS, TAAS are done through us in Japan. And we provide all or varied chain of assessments from designing, conducting, analyzing, reporting, and also operating test centers. That is our organization.

And today, John-san, I think you already know as well, I have now moved to more public role. This is the reason why I am an executive advisor to these organizations. And when I say public role, I conduct and participate as a steering committee member for Ministry of Education projects, especially in the higher education, digital transformation projects, and also global EdTech acceleration committee at Japan. My interest before was trying to influence and make an impact to the organization or my company, but now my mission statement has turned into impacting the public and society. Science of learning mission statement have evolved for me to science of system and system change mission statement.

John Keeman:

That sounds quite an interesting challenge. How do you change systems and society?

Norihisa Wada:

Well, John-san, it’s a very good question. I’m exploring that myself today.

John Keeman:

Maybe I’ll invite you back on the podcast in a couple of years and you can tell us how you did it.

Norihisa Wada:

Or I might just run away from that because I have no results, but yeah. Today, I think maybe if I explain the current situation of Japan in terms of assessment, this will help the audience understand. Currently, in Japan, like anywhere else around the world, COVID-19 had a huge impact on the government policy and the digital infrastructure. When I say digital infrastructure, meaning high bandwidth internet connectivity and one PC per child, this infrastructure in Japan in public education system suffered from low penetration in the past. And today because of the COVID-19, almost 100% penetration.

So the government policy today is how do they accelerate the use of digital contents that run on these infrastructures and how do they keep the big data in terms of advanced LMS and learning portals? And they have also provided a free to use CBT platform provided by the Ministry of Education. So when infrastructure is there, the contents are there, you can provide free assessment. And providing assessment platform is there, the government starts to want to analyze these datas as a system. And their interest today is the data standardization and how do they make that more interoperable so that they can drive better outcomes in a shorter time span so that they can be accountable for their education policy, basically the EBPM.

And this is something that is quite different from what I had been doing in the past. Nintendo education games, Nintendo console is not connected at that time, Nintendo DS. It was all in RAM. It was a standalone. But today, it’s connected everywhere. So, you have to sync this environment as a system. And how do we make this system relevant and effective? And of course in the future, there’s going to be a lot of changes, new medias, new technology, a lot of things will happen. But we need to scientifically approach how do we make a system change in a shorter time and be effective? And we have to have science behind that. So that’s the vague notion that I have today, and that’s a number one interest.

John Keeman:

And what could the rest of the world learn from Japan?

Norihisa Wada:

That’s a very delicate question you asked for a Japanese.

John Keeman:

Well, I’m sure there’s a lot that is being done in Japan the rest of the world could learn and may not know about in other language. So, any ideas or thoughts would be interesting.

Norihisa Wada:

Sure, sure. I was just joking. As a Japanese, I’m shy to say. Most Japanese will say, “It’s a delicate question because I’m shy. I cannot answer,” kind of thing.

John Keeman:

I understand, I understand.

Norihisa Wada:

Yeah, okay. What could the rest of the world learn from Japan? Well, I won’t say learn, but I would like to praise the Japanese government for their attitudes that we are seeing today, after the COVID-19 especially. Before COVID-19, the government funding for education was distributed to projects that had track record and proven record because they were worried about failures. But after the COVID-19 situation, they have changed these attitudes. One is that they started handing out funds. And some of the funds, I am a judge of selecting the projects. But they advocate that failure is not a problem since there is a system that they can retrieve data. If we are going to fail, fail fast and safe is a better way. So, they do not avoid failure today. And that’s a huge attitude change that I want to praise the government for.

And the second change… Oh, sorry. And the second change is that the government are more inclusive today. They even let student proposal projects to be funded from the government. Today, they not only advocate all stakeholders to send in proposals, but even for the judges. In the old days, judges who select these projects we’re mostly professors. But now, they invite parents, students to be one of the judges as well. So these are not being afraid of failures and also becoming more inclusive, I think this is two points that I would like to praise the government.

John Keeman:

Yeah. No, that’s very interesting because I think a lot of governments and education bureaucracies are scared of failure. And also, I think we’re seeing challenges all over the place about people trying to be more inclusive and working out how to be. In terms of inclusivity, any suggestions or advice on how assessment organizations can be more inclusive?

Norihisa Wada:

Sure. That is a very good question today because the EBPM, evidence-based policy management, is a huge concern for the government and interest for the government. Assessment industry plays a huge role here. And there’s many assessment organization, not only in Japan but worldwide, who can give enlightenment to whatever the projects that the government initiates. And the government are more willing to learn from not only Japanese organizations like us, but worldwide as well.

John Keeman:

And it also sounds like Japan is trying to follow the evidence and follow the science. I suspect that’s something that other organizations are also trying to do, but maybe not always succeeding.

Norihisa Wada:

I think they have more interest in science of system and science of system change today.

John Keeman:

Interesting, interesting, interesting. And maybe we’ll have you back, like we said, to see how that’s going. Let’s move on to a final subject area, the future of assessment. Will test and exams be the same in 5, 10 years’ time? How do you see things changing?

Norihisa Wada:

I think future is likely not test and exams as we know them, but people being assessed or measured continuously anywhere and everywhere. I have seen new technology, AI, voice recognition, and new media like Nintendo DS impact learning and measurement in my past career. So I expect the same will always happen in even shorter cycle in the future. In this sense, I am curious to see how new media like metaverse and new technology, XR, AR, VR will impact assessment.

John Keeman:

So that maybe people can be in the metaverse and do things in the metaverse or with virtual reality and be assessed without maybe knowing about it.

Norihisa Wada:

Right. That is exactly right. Like I mentioned, stealth measurement, stealth assessment. So, whether will we be able to utilize these new technology and media in a fair, valid, and interconnected way and improve our current system? And these are the projects which I like to explore in the near future. And circling back to metaphor of Mario Kart that I used, I guess my ultimate goal is to create a Mario Kart of learning, assessment, and navigation system where every stakeholder, or should I say players, is improving. Players is improving personally and also while contributing to the improvement of the system change and enjoying such process, having fun in the process.

John Keeman:

That’s a very interesting vision that we might all be players rather than test takers and just be assessed as we go. Thank you, Nori-san.

Norihisa Wada:

Thank you.

John Keeman:

And thank you, our audience, as well for listening with us today. We really appreciate your support. And don’t forget, if you’ve enjoyed this podcast, why not follow us through your favorite listening platform? And also, please reach out to me directly at john@questionmark.com with any questions, comments, or if you’d like to keep the conversation going. You can also visit the Questionmark website at questionmark.com or the Learnosity website at learnosity.com to register for our best practice webinars and see other resources. Thanks again and please tune in for another exciting podcast discussion we’ll be releasing shortly.

Related resources

Get in touch

Talk to the team to start making assessments a seamless part of your learning experience.