John Kleeman is joined by assessment luminary, Mike Olsen, the CEO and Founder of Proctorio. Mike discusses what drove him to create a company focused on proctoring, how it has expanded beyond the education sector, and how the tool can be used to address test fraud.
Full Transcript
John Keeman:
Hello everyone, and welcome to Unlocking the Potential of Assessments, the show that delves into creating and delivering and reporting on fair and reliable assessments. In each episode, we chat with assessment luminaries, influencers, subject-matter experts, and customers, to discover and examine the latest in best-practice guidance for all things assessment. I’m your host, John Kleeman, founder of Questionmark, an EVP of industry relations and business development at Learnosity, the assessment technology company. And today, really pleased to welcome Mike Olson, who is CEO and co-founder of Proctorio, an online proctoring company that uses machine learning and AI technology to validate student activity, and identifies the right people in computer-based learning environments.
Mike was dubbed an artful hacker by PC Magazine and named one of Arizona’s 35 entrepreneurs under 35 by The Arizona Republic. He’s a leader in creating educational technology that protects student privacy, while increasing the value of online degrees. As lead software engineer for ASU Online, Mike helped develop one of the world’s premier online educational systems, and I think graduated from Arizona State University with degrees in aerospace and software engineering. Mike has a reputation for disrupting all paradigms, and increasing student success in digital classrooms. Welcome, Mike. Really pleased to have you here.
Mike Olsen:
Thank you, John. Really excited to be here with you.
John Keeman:
Let’s start as we usually do in this podcast, again asking how you got into the world of assessment.
Mike Olsen:
My background is in education, so higher education mostly. I used to work at Arizona State University as part of what was called ASU Online at the time, and now is called EDPlus. And it’s actually real funny. I was hired on very, very early in that program. My first day was interesting. They had just fired everyone who had worked there. A new leader had taken over, and they were basically going to modernize and clean house. And so I was hire number six in this new group. We had this small hallway that they put us in, in the back of this thing called the computing commons. It was really, really creepy. But we started, and at the time, the six of us, there was maybe a total of 40 online students at the time. And when I say online students, I mean these are fully-online students.
And through that process, I was there a total of six years. And by the time I left at Year 6, we had 10,000 fully-online students, and these are students who maybe have never even been to campus, maybe have never even been to the State of Arizona. They’ve signed up online, they’ve taken courses online, and they’re graduating even online. On my last day there, we actually announced the ASU partnership with Starbucks, which gave baristas the opportunity to go to ASU Online, it was covered by Starbucks, and further increased that program. So, it was a lot of fun. And the way I got into testing, is if you think about it from 40 to 10,000 students, there’s a lot of testing along the way. Most students, on average, two and a half assessments per course, most students are doing six to eight courses per semester, and two semesters a year. And so with that, a lot of assessment going on in education.
John Keeman:
I believe you were dubbed the artful hacker by PC Magazine. Tell us about that before we get into the meat of the assessment side of things.
Mike Olsen:
I have a copy of the magazine somewhere, and I think there’s an online version of it as well. I was really into… And this is going to date me, because LAN parties, they were a thing. We didn’t really have internet, maybe dial up, but no one was connecting online. So everyone would grab their computers, we’d go to a friend’s house, and we’d play games. And I really like that, but then somehow I got into stats, and collecting stats, and analyzing stats, and that’s really what got me into coding. And what we did, is we built this game dashboard.
I don’t remember exactly how we did it, but we submitted it to PC Magazine, and then they printed it, and took a photo of us. It’s very old. It’s very funny. But we’re there with our custom computers and this dashboard, that we’ve got this piece of wood, and we’d hand painted the text on it. So it was this blending between artful, I guess, with the dashboard and designing it, and then hacker, hacking the games so that you could get the stats from it, so you could analyze your stats and things like that. So that’s how that happened.
John Keeman:
Cool. So, tell me about how you founded Proctorio? What was the need that you saw there, and how did you do it?
Mike Olsen:
I think the most important thing that a lot of people, I think know but don’t realize, is… We saw it at ASU. The number one inbound question for us on a student who’s thinking about… or a prospect becoming a student, was, “Does my degree say online? I’m going to ASU. I want a degree that says ASU on it, Arizona State University, but does it say online? Will my employer basically know I did it online?” And the reason that question is asked, is because there’s this perception that an online degree means less, or means they cheated, or means that it was easier. And I got into it, because we at ASU, we had decided, “Look, there are certain ways to validate that.” And it’s basically, when that person is signing up, and then throughout their entire educational career, we’re validating that it’s the right person, we’re validating that they’re not cheating. In the assessment periods, we’re proctoring or monitoring them.
And what we used at the time was, I would say, not technology. Now, this was a while ago, but it wasn’t technology. We were having TAs initially connect over Skype, and they would Skype session with a student. That works when you have 40, but it doesn’t work when you have 10,000. And so we started outsourcing it to these companies. And then these companies would basically charge students a lot of money. They’d have scheduling. So you had to book a certain slot. And if you think about it, at the end of the calendar year in December, final exams occur, at least in the US, occur over a two-week period. That two-week period to get a time slot, was nearly impossible to get something ideal.
So you’re talking about a student that you’ve sold saying, “Hey, it’s online education. Take it whenever you want. Do it from anywhere. Oh, except during finals, because you’re going to have to sign up for a slot. It’s probably not going to be convenient, and the company making you do it, is going to make you take it in a certain environment.” So, it wasn’t a good experience. At ASU, we wanted to create something… And I was the software engineer, actually the only software engineer on staff, that wanted to create something, and eventually was told no, and I decided to quit and create a remote proctoring company, that I felt could solve those issues. So, no source scheduling. It was completely automated based and respected the privacy of those individuals.
John Keeman:
And when was this? When did Proctorio start?
Mike Olsen:
2013. It feels like a long time ago. We’ll hit 10 years in April actually.
John Keeman:
How many people do you employ now?
Mike Olsen:
We have 140 people across three offices. Our primary headquarters, which I’m in right now, is in Scottsdale, Arizona. And then we have two offices in Europe, that handle the rest of the world.
John Keeman:
And I think one of them is in Belgrade. What made you choose there as a partial base?
Mike Olsen:
Belgrade is interesting. One thing a lot of people don’t realize, is although we’ve created a proctoring business that uses automation… We don’t employ proctors; we use software and technology that does it. But one thing a lot of people don’t realize, is that even though we have essentially robots as proctors, we have a lot of people who actually are what we call Tier 1, the first responders. The people who help the test takers, no matter where they are, get into an exam. A lot of people finally experienced this during COVID, cameras don’t work, microphones don’t work, there’s technical issues all the time. We need people that actually help those test takers get into exams. And one of the things that we’d realized, is we needed some contingencies. We needed a scalable operation that had people in multiple time zones across the world, people who spoke multiple languages.
And then with GDPR… so this was pre-GDPR days… GDPR around the corner, we needed people located in Europe, who could handle those sorts of conversations, and that’s through custom data. So Belgrade was one of the areas we were looking at. There’s a lot of development… I don’t know if you’ve heard of a company called NCR, but NCR has a huge facility there for support, sales and engineering, and so we wanted to copy that model. We’re actually right down the street from Microsoft. Microsoft has a huge office there in Belgrade. It’s actually where the HoloLens was created. The first time we took a tour of that facility, they were just ripping the plastic off the windows that they had covered, where they were doing the HoloLens development secretly behind those windows. So Belgrade answered a lot of questions for us. We initially tried it with just the third shift or things like that with support, but now we have the dual-support operations, both here in the US and over in Belgrade.
John Keeman:
And what do you think the proctoring revolution gives people that helps… Is it just really online education in universities, or is Proctorio also used in other places?
Mike Olsen:
Look, it’s probably been said a million times, and I’m going to say it again, but COVID definitely changed our industry. I think that’s assessment, that’s real proctoring. Pre-COVID, there were a lot of people who thought it had to be in the classroom, it had to be in the testing center, and then when those options went away, a lot of industries, not just education, finally figured out that, “Wait a minute, I don’t have to force someone to drive or travel, stay in a hotel, go take this exam at this secure facility, and that’s more secure than them doing it at home.” The realization had finally kicked in. Remote proctoring, the revolution, I think comes down to people who… When we initially started, Proctorio was used as, what we call an accommodation. There are a lot of test takers out there… pre-COVID especially… a lot of test takers out there, who didn’t have the opportunities that the traditional student had.
Think about your working mothers. Think about people with disabilities, whether that’s physical. Think about people who have a certain career, or path, or live somewhere, the rural student. These people who are trying to get a better education, trying to improve their skills, go get a certification, they didn’t have a voice in this education concept. They couldn’t participate. And so remote proctoring has really enabled a large group of people throughout the world to join in and start taking assessments in a trusted with integrity way at a university, or with certification. So, I would say the biggest thing, is there was a group of people who didn’t have the opportunity, who now do. And now that this world has changed and acknowledged and said, “Look, there are different ways to do this there. There’s still maintain the integrity, we still maintain all of the things that we think that a testing center’s going to do. We can maintain that, so let’s give these people an opportunity.”
And so, we do strange things. We certify firefighters, we certify these oil rig workers, coal miners. These were people who, if you’re on the oil rig and you have to take hazard safety training, what happens is you would have to fly, or take a boat, go stay in a hotel, take the assessment at a testing center, and then go back, just to certify that you knew how to operate machinery safely. These people are very highly skilled, very, very expensive, and they would lose an entire week. Now they can do it onsite. People are maintaining their hazard safety training, and it’s better for everybody.
So I think the revolution is just starting, but you have a group of people who either were not served at all, pre this remote proctoring change, and then you have a group of people who had to deal with a lot to go get these sort of things that should have been easier. So I think this is just the beginning of it, but we’re seeing a big change throughout… Not just education. I think education was one of the first ones that embraced the remote proctoring, but we’re seeing it throughout all industries.
John Keeman:
No, we’re suddenly seeing a lot of it as well. So one of the things that surprised me about Proctorio, is that I think your security and your privacy is very strong, but you tend to get attacked quite a lot in social media and things. And people like to almost throw darts at you as being the big-bad people in proctoring, when I think you actually have a very good story. So I mean, tell me a little bit about the security and privacy of Proctorio. Can you, for example, see the videos that you record? Can any of your team?
Mike Olsen:
Look, that’s what makes a big difference. I’ve always wanted to take approach to, how can you do what the other guys do, but do it in a way that respects the privacy of others? And so we’ve taken that approach very, very early on. We’re the only company in this space that uses end-to-end encryption, and that’s not just between transferring video, that’s actually between the test taker and the organization that’s administering the exam. I know it sounds like a bunch of jargon, but it means exactly what you just said. No one in this organization, so not a system admin, or even… So, we use Microsoft Azure or AWS, right? No one at those companies even, can just log in and start viewing recordings. No one in this organization, no one in any of the providers that we have, these third parties, can do that.
That’s not true with anyone else in this space. Anyone else in the space, there’s definitely an IT person, a CTO, someone who can log in and go view these sorts of things. Now if you think about it, we knew early on these recordings would be captured in things that are very, very private, such as the bedroom. There’s a lot test takers, who the only quiet place they can take the exam is their bathroom. These are personal spaces that shouldn’t be available to just anybody, shouldn’t be available to the third-party proctor or vendor. And so very, very early we said, “Look, we have this opportunity where we can remove the human proctor from the situation, and just make it between the test taker and the administrator.”
So, it’s no different than a proctor center, a testing center. It’s the same thing, the relationship between the two. They’re just doing it asynchronously, and in a location, and on a device that is their own. So I think it makes a big difference. And that goes all the way through the company. So for example, we were the first ones… I know a few have copied us since. We were the first ones to use single sign-on. So, we’re not collecting usernames and passwords. Our entire communication system with support, is actually pseudonymous too. We don’t get emails, we don’t get names. None of that stuff comes through. Now, it bites us sometimes, because the school will say, “Hey, Johnny said he took an exam, contacted support, can you show us what he said?” And we have to come back and say, “No, we don’t know who Johnny is. We can look at all the chats for your accounts during that day, and maybe you can figure out who Johnny is,” but we don’t even capture it there.
So it makes a big difference, especially today with data breaches and things like that. If someone were to breach our systems, they’re going to get a bunch of stuff that has pseudonymous identifiers, encrypted recordings that we can’t decrypt. So it makes a big difference for the consumer. We bring this all the way through the organization. If you go to our website, there’s no cookies. I can guarantee we’re the only one in the space who has no cookies on their website.
John Keeman:
Wow.
Mike Olsen:
So, no cookie banners to accept, which we’ve gotten, “Hey, you don’t have a cookie banner. You’re in violation.” It’s like, “Well, actually there’s no cookies, so we don’t need a banner.” We do this throughout the entire organization. I would say it’s a methodology, it’s something that the company was founded on, and we’re able to do it. I mean, look, by not having the proctors, by using automation, by using technology, and by doing this end-to-end encryption, we’re able to take something that isn’t normally very private, or privacy focused, and make it as privacy focused as possible for the consumer.
John Keeman:
And so just to clarify that, if I say, take an exam using Proctorio with, say a university, the university can see my recording if they want to check that I took it honestly, but nobody at Proctorio from you down can see it, and it’s impossible for you to do so, because it’s protected by encryption and things?
Mike Olsen:
Exactly. The best we could see, is that this unique ID took an exam that was this long. Can’t see any of the other metadata, can’t see any of the recordings themselves, or if they require an ID capture. Think about that. All the other vendors are capturing, maybe driver’s licenses, or passports, or things like that, and that’s available to that vendor. So it’s just like you described, only the customer, the exam administrator, can decrypt those reportings.
John Keeman:
And what about if the computer flags cheating or possible cheating? Can the computer using Proctorio stop an exam? Do you think it ever should, or could, or would like to in the future?
Mike Olsen:
So, this is something we get fought on all the time, especially as we enter the spaces outside of higher education. Our opinion has always been that the computer should not take an action. So maybe it’s a problem with certain industries, but in our opinion, look, the computers can do very simple black-and-white analysis, but what is cheating? I’m sure you have a different definition than I do. I’m sure that on a per exam, or per test, or per course, or per certification basis, there’s different considerations whether something is cheating or not. There are different types of people, people who need things like screen readers, or need people that actually sit there and help them read or write out the exam, or understand it, translators, things like that. So anyone who’s selling this snake oil, I would say, where it’s this, “Oh, we can detect cheating, and we can stop them in the exam,” and things like that, it’s simply BS. I don’t believe it, because what is cheating, really, really depends on the circumstances.
So Proctorio has always made the decision that we don’t make a decision like that during the exam. So we will not remove a test taker using algorithms, or AI, or things like that from the session. What we will do, is simply flag every type of event, multiple people, looking away, getting up from the exam, opening new browsers, opening new programs. We will flag those things, and deliver that report to the exam administrator. And they’re the ones who can decide whether they even care or not. They can even decide on a per test taker basis. So it’s not like there’s a set settings on the entire exam. They can actually go in and say, “Hey, this person has an exception.” Maybe they have IBS and they need six bathroom breaks. So that exception can be applied, and that goes into that analysis. These kinds of things, I don’t think computers should be making the decision in. I also don’t think third-party proctoring or other vendors, should also be making the decision. This is between the test taker and the exam administrator.
John Keeman:
I know another big issue is whether AI-type algorithms are fair to everybody, and deal with people with different demographics and things. How do you approach that to try and be as fair as you can do?
Mike Olsen:
It’s something that a lot of people are finally talking about, which is good. I think some of the first conversations that have happened over the last couple of years, are really differentiating facial recognition versus facial detection. There are different products there, and a lot of things that have been found to be biased are based on recognition products, like AWS recognition, and things like that. And what we’re actually learning now, actually we’ve known for a while, but what the world is finally learning, is that algorithms, it’s garbage, garbage out. So if you don’t take care and really adopt a methodology for training algorithms, you’re going to get terrible data out. And so for the last several years, since the beginning of the company, we’ve trained algorithms based on sets of data that we try to get a variety of… think about skin tones, genders, ages, things like that… so that we could remove or eliminate as much bias as possible.
Over the last couple years, we’ve actually adopted an auditor who goes through, and every time we do a model change, they go in and take groups of data that our system has never seen with known outcomes and test it against that. And so, it’s a process. It’s a process of continuing to update the algorithm, continuing to check it, and look for these sort of biases that could be introduced. And that process just keeps improving and improving and improving. We’ve adopted a new program here. We’re doing all kinds of, I would say cool things, where all of our AI is sourced off of ethical data that’s been collected. So we’re not stealing data from users or customers or things like that. It’s been ethically sourced. The other thing we’re doing too, is all of our training now is done on sustainable energy.
It’s not a sunny day, but we’re lucky here in Arizona. It’s mostly sunny. We installed a huge solar farm, and all of our models are actually being generated on solar power today. So, it’s a big difference. I would say the big differentiator for us, is we’re building our own models and our own products because we can test it, and we can continue to iterate on it. A lot of people are just using things out of the box, like Amazon recognition and things like that. Actually, I think everyone is using that. And that comes with a lot of problems. I mean, here in the US it’s banned by every police department to use things like that.
And so why that’s being used in education, is kind of shocking to me, but I think… Look, the criticisms are great. The only way companies like Proctorio can continue to improve, is by receiving criticisms, evaluating them, determining whether or not they’re our product, or just a general understanding of the market and things like that, and taking action on that. And so for a long time now, we have been doing what we consider ethical-based AI. We’re going to continue to do that. And we have a few more things that you’ll see this year that we’ll release, but it’s the right approach to doing this.
John Keeman:
I think I know your answer to this, but what’s your view on human against AI proctoring, was that people who abdicate human proctoring suggests that it’s better quality at identifying cheating, and also that for a lot of AI proctoring, people don’t always review it as much as they should do?
Mike Olsen:
So look, my opinion, I think, comes down to one thing, and it’s the most important thing, and it comes off the previous question of biases. Here’s the thing that computers are very good at. Computers are very good at being consistent. So if I were to do an evaluation of the algorithm, and go through… And we’re always improving it, but our latest reports say there’s no statistically significant bias. The computer will always nail that. The computer will consistently produce those results over and over again, and we can continue to improve that. What I say to the people who use humans, is there is no way you could ever evaluate your human staff of proctors and get consistent results.
People have bad days. People will start their shifts, and they will have a bad day. People are tired. You never know what’s going on in someone’s life, and then they bring that to work. We talk about algorithms being biased because of people, well, people are biased. So, humans are much worse at things that I would consider are binary, which is evaluating whether the person, for example, is there, and they’re in front of the camera, and they’re working on the exam. Humans make all kinds of assumptions. Humans also have things like background checks that should be required. Well, the computer doesn’t need a background check. I’m pretty sure the computer isn’t a felon, and those sorts of things happen.
And so my opinion on the humans, is look, humans still are involved in the process, like I said earlier. We have humans who help the test takers get in the exam. That’s something really only a human can do. And then we have humans who evaluate it at the end. I would say certain clients of ours will use it to check a box and they never review the recordings. Sometimes that works. I don’t know where you are you have the cameras on the road that are taking pictures when people are speeding. You never know if that thing’s plugged in. And so, there’s obviously a sort of factor there that is considered. Just having the camera on the side of the road, are we reducing speeding? So I think by having the proctoring software there, we’re reducing the cheating.
The other thing too, that I don’t think a lot of people talk about when it comes to whether they’re reviewed or not, is most test takers will end up cheating because they think someone else is cheating. They’ll think that the playing field is not level. And if you do any of the student surveys like we do, all of them think that when the proctoring software… They feel that the playing field has been leveled. They feel that they have the same advantages or disadvantages the students, or the rest of the test takers taking that same exam do. And I think that makes a big difference. Those are the sort of things that matter. So whether you review them or not, you’re still putting something in place that is putting a level of integrity in there. So I think, look, humans are involved in the process certainly, but for the things that humans are not good at, which is watching people for eight hours a day, I don’t think you should be doing that.
I think people are a big problem. And I think of all the issues that we had seen in our industry over the last few years, most of them you could trace back to humans. There was an example in our industry, the proctor made someone pee in a cup. I can guarantee you, my software doesn’t have any code in it that says, “Okay, your exam is paused, go pee in a cup. You can’t leave the session.” So, humans can make all kinds of crazy decisions, absurd decisions. Humans can also, like I said, just have a bad day. And based on that, I would say that the AI, at least the way we do it, where we blend AI and human, is superior to a human-only approach.
John Keeman:
What about content theft concerns that a lot of people have? Because obviously, if you don’t stop the exam during the exam if you find somebody copying content… What’s your thought on that?
Mike Olsen:
Well look, it’s a problem that’s existed for a long time. Even with physical exams, you had theft of the exam items, even though they were using secure carriers, and police even guarded material. So, theft is always going to happen. I think, look, AI proctoring means as technology based, at least with our company, we’re not using products like Zoom or GoToMeeting or things like that, and calling it automated proctoring, we actually use technology. And the technology itself, through a combination of watermarking that we’re doing to the screen, a combination of the exam security, so blocking screenshots, blocking recording tools, things like that, you’re eliminating most of the ways that people are going to steal content.
The other ways that they’ll steal content… They’ll still do, but what we offer, and I know a lot of people… There’s entire companies actually dedicated to this. What we offer, is sort of take-down services and discovery services. So we can identify if stuff has leaked, to what extent it’s leaked, and whether we need to change content or things like that. We’ve got a product, at least in higher education, which can identify any stolen content, and actually rotate those questions out, so they’re not used anymore. So, there are ways to battle that.
John Keeman:
Oh no, sure. And I think the dynamic content that changes, is increasingly being adopted. Before we run out of time, I would like to get your views on the future of education and assessment. Where do you think we’re going?
Mike Olsen:
So, look, I think the most important thing is education itself is becoming more and more personalized. It’s becoming more and more available. There are certain institutions that people could only go to because they were elites, or they lived physically in a certain area, and that’s changing, and for the better. And so there were things like MOOCs, and I don’t know if that formula worked, but education, or access to education, globally is becoming easier and easier. And I think what changed in education at least, is this digitization, which was a huge thing that had to occur, and this acceptance of the remote or the non-physical person on campus. And I think the entire industry when it comes to assessments, whether it’s in education or certifications, is going through a revolution.
Look at work from home, for example. People are not comfortable anymore going, parking, driving, going to the testing center. The testing centers are invasive. You have to basically strip down, you put everything in a locker, they pat you down, you go through a metal detector, something that’s more invasive than airport security. People are not comfortable with that anymore. And it only hurts a population that’s trying to become educated. I think we would all agree that the more education people can get, the more educated we are, the better humans we’ll end up being, the better the future generations will be. And so any of these sorts of barriers or burdens that we currently have, eliminating them, dropping them, and making things easier, is the future. And the thing that… Higher education has started it, I think certifications are going to adopt it very soon. Is, “Can we still do this and respect the integrity?”
Think about it. You’ve got bridge builders. You don’t need bridges collapsing. You have nurses. You don’t need them killing people. You have pharmacists who should be issuing the right drugs. There’s a level of integrity that people expect, and now that there are tools like Proctorio or any of the other remote proctoring tools, people can say, “Okay, does my degree say online on it? No, it doesn’t, but it doesn’t matter.” And what I’m hoping, is that we can actually switch to this paradigm to say, “I want to go online, because it’s more personalized, because it’s more convenient. I can maybe move through the education system quicker.” And that’s what I would like to see. So I think the future is the opposite of what I saw 10 years ago. Not the concern that it’s online, but actually the demand or the request, “Can I get an online degree? Because it’s going to fit my needs, and it’s going to be convenient for me, and the world will be a better place.”
John Keeman:
That sounds great and a good place to end. Thank you very much, Mike. And thank you all our listeners for listening for us today. We appreciate your support. And don’t forget, if you’ve enjoyed this podcast, why not follow us through your favorite listening platform? We have a new episode every month. And also, please reach out to me directly at john@questiomark.com with any questions, comments, or if you’d like to keep the conversation going. You can also check out more about Proctorio at Proctorio.com, and visit the Questionmark website at Questionmark.com, to register for any of our best practice webinars that we host frequently. Thanks again, and please tune in for another exciting podcast we’ll be releasing shortly.
Mike Olsen:
Thanks, John.
John Keeman:
Thank you, Mike.