New research from the University of Toronto’s CitizenLab pulls the curtain back on how WeChat’s real-time, automatic censorship of text and images is used to exert control over political discussion on topics ranging from international issues like the trade war with the US to domestic scandals like the disappearance of court documents in a 2018 dispute between two multibillion-dollar Chinese mining companies. All discussion is ultimately subject to the Chinese government’s approval.

WeChat’s censors face two types of challenges. Big public posts on WeChat Moments, a public feature similar to Facebook’s Timeline, are scrutinized and filtered by algorithms that can sometimes take over 10 seconds to run—a glacial pace on social media. But one-on-one and group messages are a different problem entirely, because they are often intimate and instant conversations. That requires real-time censorship.

Text is relatively easy to search and censor. Image filtering is harder, especially when you’re trying to examine and censor the images almost instantly. To accomplish this task, WeChat keeps a massive and always growing index of MD5 hashes, small cryptographic data signatures that are unique to every file. When a censored image is sent, it will be caught by the hash index and deleted. Neither the sender nor the recipient is ever likely to know anything was censored.

If the image isn’t instantly censored, it’s sent for automatic analysis. Using optical character recognition, the image is examined for text (sending screen shots of newspaper articles was once an easy way past censors). The image is then checked for visual similarity to other censored images. So-called harmful content—including anything about international or domestic politics deemed undesirable by the Chinese Communist Party—will be sniffed out, removed from the conversation, and then added to that original hash index, which flags it for instant censorship from that moment onward. It’s a self-reinforcing system that’s growing with every image sent. The latter systems are also used on WeChat’s Moments to check and build the company’s dynamic blacklist.

The Chinese tech giant Tencent, which owns WeChat, is under “a great deal of pressure” from Beijing to implement effective censorship tech, says Adam Segal, the director of the digital and cyberspace policy program at the Council on Foreign Relations. “The Chinese firms are all responsible for content, and while they have relied on tens of thousands of human censors, they have also been developing new [machine learning] approaches to content take-downs. Inside of China, it is part of the larger trend under Xi Jinping of tightening controls on the Chinese internet and society more broadly.”

WeChat is so pervasive in China that the prospect of getting suspended or banned can disrupt lives. The app combines the features of Facebook, Uber, GrubHub, and more. You can book doctor’s appointments, pay utility bills, talk to professional contacts, or engage government services.

“This has really become a mega-app,” says Sarah Cook, the senior research analyst for East Asia at the pro-democracy research group Freedom House. “It’s really hard to function in modern Chinese society without using WeChat, and so the chilling effect is real.”

Under the direction of Beijing, Tencent’s censors are adroitly responsive to current affairs. In addition to deadly historical events like the Tiananmen Square protests and the Cultural Revolution, new events are quickly added.

Researchers found that newsworthy events like the arrest of a Huawei executive on charges of fraud and the CRISPR-baby scandal in which a Chinese scientist announced the birth of girls with edited genomes triggered waves of censorship. So did a host of other issues, like conflict with the US and domestic corruption or other problems.

One event triggered even more censored images: in December 2018, a scandal erupted when the Chinese Supreme Court admitted the disappearance of documents in a billion-dollar dispute between mining companies. Researchers found that dozens of images relating to the alleged theft of the documents were censored.

“In the last few years, you’ve had the combination of tightening censorship restrictions and tightening controls,” Cook says. “There’s less official tolerance for conversation, expression, and networking that had been tolerated previously. You’ve seen Tencent and WeChat, in particular, get hit with fines. So now they’re trying to employ new methods to catch so-called ‘harmful content’ that the Chinese Communist Party wants to stop. That’s where real-time automated censorship comes in.”