Despite arriving too late to prevent what some might call a full-blown crisis, Overwatch’s anti-toxicity efforts seem to be bearing fruit. The next step in the battle, says game director Jeff Kaplan, might be having computers, not humans, find the bad actors.




“We’ve been experimenting with machine learning,” Kaplan told me during an interview at Blizzard’s offices yesterday. “We’ve been trying to teach our games what toxic language is, which is kinda fun. The thinking there is you don’t have to wait for a report to determine that something’s toxic. Our goal is to get it so you don’t have to wait for a report to happen.”

Even without robots leading the charge, Blizzard is making progress. In a video released in January, Kaplan said that thanks to changes to the game’s reporting and punishment systems, abusive chat went down by 17 percent. He claimed that people use the report feature 20 percent more often, too. So basically, Overwatch’s reporting system is functional now. But that doesn’t necessarily mitigate the problems shared by reporting systems as a whole. They’re slow. People don’t always have faith in them. They can be abused. The next step for the Overwatch team, then, is to start catching bad behavior before it’s ever reported. Enter machine learning. Already, Kaplan says, the company is teaching its AI non-English languages like Korean, and the long-term hope is that it can learn what toxic gameplay looks like.




“That’s the next step,” said Kaplan. “Like, do you know when the Mei ice wall went up in the spawn room that somebody was being a jerk?”



“Our goal is to get it so you don’t have to wait for a report to happen.”

Of course, machines are just as fallible as the humans who built them, albeit in different ways. Machine learning can have trouble determining context, especially if you’re just teaching it words and phrases. Profanity-laden trash talk, for example, might cause you to see red and spit fire if it’s coming from a complete stranger, but laugh like an idiot if it’s coming from one of your best friends.



Kaplan said that, at this point, the team has mostly tuned its AI to go after the most egregious examples of toxicity—people who, by Blizzard’s metrics, doubtless deserve punishment. It’s also not going through DMs between friends or anything like that, and it can be tuned to flag first and only punish on repeat offenses.




“With anything involving reporting and player punishments, you want to start with the most extreme cases and then find your way to ease up after that,” he said.



The Overwatch team—which is now part of a Blizzard-wide anti-toxicity initiative as well as a collaboration involving companies like Riot and Twitch called The Fair Play Alliance—is also examining the other side of the coin: how do you reinforce good behavior?




“We can start looking toward the future and talking about things like, what’s the positive version of reporting?” said Kaplan. “Reporting is saying ‘Hey, Adrian was really bad and I want to punish him for that,’ but what’s the version where I can say ‘That Adrian guy was an awesome teammate and I’m so glad I had him’?”



“We’re punishing the bad people, so how do we get people to start thinking about being better citizens within our ecosystem?”