For the last few years, police forces around China have invested heavily to build the world's largest video surveillance and facial recognition system, incorporating more than 170 million cameras so far. In a December test of the dragnet in Guiyang, a city of 4.3 million people in southwest China, a BBC reporter was flagged for arrest within seven minutes of police adding his headshot to a facial recognition database. And in the southeast city of Nanchang, Chinese police say that last month they arrested a suspect wanted for "economic crimes" after a facial recognition system spotted him at a pop concert amidst 60,000 other attendees.

These types of stories, combined with reports that computer vision recognizes some types of images more accurately than humans, makes it seem like the Panopticon has officially arrived. In the US alone, 117 million Americans, or roughly one in two US adults, have their picture in a law enforcement facial-recognition database.

But the technology's accuracy and reliability at this point is much more modest than advertised, and those imperfections make law enforcement's use of it potentially sinister in a different way. They're prone to both false positives—a program incorrectly identifies Lisa as Ann—and false negatives, in which a person goes unidentified even if they're in the database.

Bad Reads

For an extreme example of what can go wrong, take data recently released by an EU Freedom of Information request and then posted by the South Wales police. It shows that at the Champions League final game in Cardiff last year, South Wales police logged 173 true face matches and wrongly identified a whopping 2,297 people as suspicious—a 92 percent false positive rate.

"From a government’s point of view a dragnet that catches a lot of extra people from which they then filter out what they’re interested in might be considered as working and might not cost them too much," says Suresh Venkatasubramanian, a professor of computer science at the University of Utah who studies discrimination and bias in automated decision making. "But from your point of view if you’re caught up in one of these false positive dragnets, that might not seem like it's working to you."

The South Wales police department says it has refined its algorithms and improved the quality of images in its databases since then, but it still describes that early deployment as successful. "The past 10 months have been a resounding success it terms of validating the technology, building confidence amongst our officers and the public whilst offering a potential area for growth for us with the technology in the future," the department wrote in a defense of its facial recognition program. Meanwhile, according to its own data, the South Wales system had an 87.5 percent false positive rate at an Anthony Joshua boxing match in Cardiff at the end of March.

"Normal error rates would suggest that you’re going to get a lot of hits if you just indiscriminately take a lot of people’s faces and run them against your database, versus the other way around where you target your search for a specific person and try to match that person’s face to the crowd," Venkatasubramanian notes. "There are subtleties in how a system is deployed versus how it was trained. We often see a failure mode for the use of algorithms in these systems where they’re trained for one thing, but they’re being used a slightly different way and that causes problems."

'If you’re caught up in one of these false positive dragnets, that might not seem like it's working to you.' Suresh Venkatasubramanian, University of Utah

Those worried about how facial recognition surveillance may impact their personal privacy may view these flaws as a potential advantage; a fallible system might be easier to hide from. But in practice, these deficiencies can cause innocent people to be flagged as suspicious and can even lead to wrongful arrests. In one example in the US, a Denver man was arrested two separate times in connection with two bank robberies perpetrated by someone who looked like him in CCTV footage, but was later determined to be someone else. Researchers have also found that societal biases, such as racial prejudices, are reflected in the data used to train facial recognition models, and in the algorithms themselves.