The US government is once again reviving its campaign against strong encryption, demanding that tech companies build backdoors into smartphones and give law enforcement easy, universal access to the data inside them.

At least two companies that sell phone-cracking tools to agencies like the FBI have proven they can defeat encryption and security measures on some of the most advanced phones on the market. And a series of recent tests conducted by the National Institute of Standards and Technology (NIST) reveal that, while there remain a number of blind spots, the purveyors of these tools have become experts at reverse engineering smartphones in order to extract troves of information off the devices and the apps installed on them.

Asked whether the NIST test results have any bearing on the public debate about backdoors for police, Barbara Guttman, who oversees the Computer Forensic Tool Testing program for NIST told Motherboard, “None at all.”

“This is a completely different question. That’s a policy question,” she said, adding that NIST’s only purpose is to ensure that “If you’re acquiring the phone [data], you should acquire it correctly.”

But the demonstrated ability of phone cracking tools to break into and extract data from the latest phones is further proof that the government is perfectly capable of getting into terrorists’ devices, Andres Arrieta, the director of consumer privacy engineering at the Electronic Frontier Foundation, told Motherboard.

“When it comes to the capabilities from law enforcement, I think these documents show they’re quite capable,” he said. “In the San Bernardino case, they claimed they didn’t have the capabilities and they made a big circus out of it, and it turned out they did. They’ve proven consistently that they have the tools.”

The never-ending public debate over smartphone security has focused on backdoors for law enforcement to bypass device encryption—and more recently, Apple features that erase all data after 10 failed password attempts or block data extraction through lightning ports. But accessing a phone is only part of the battle; once inside, digital forensic investigators have to understand the complicated data structures they find and translate them into a format that meets the high accuracy standards for evidence, using acquisition tools from companies like Cellebrite, Grayshift, and MSAB.

Results from an NIST test of Cellebrite found that it largely works as expected.

In a series of reports published over the last year, NIST’s Computer Forensic Tool Testing program documented how well the latest tools perform that task on dozens of different smartphones and apps. The tests paint a picture of an industry trying to keep pace with the constantly changing smartphones and social media landscape—with mixed results.

“Let’s say you can get into the phone, you can defeat the encryption. Now you have a blob of ones and zeros,” Bob Osgood, a veteran FBI agent who is now the director of digital forensics at George Mason University, told Motherboard. Smartphones contain millions of lines of code, the structures of which differ between every device and can change with every OS or app update. Cracking a phone’s encryption doesn’t necessarily mean an investigator can access the code on it, including deleted and hidden files, hence the need for the tools tested by NIST. “In the digital forensics world, the state of complete Nirvana is to get a complete image of the phone,” Osgood said. “The amount of technical know-how it takes to actually do this stuff—reverse engineer, beat the encryption, get data itself—is massive. There are a million moving targets.”

Take Cellebrite, the Israeli company whose Universal Forensic Extraction Device (UFED) is a favorite of police departments and the FBI. In June, the company announced that its new premium tool could crack the encryption on any iOS device and many top-end Androids—a major win for law enforcement agencies that had been complaining about built-in encryption.

The company’s current UFED 4PC software is then capable of accurately extracting the vast majority of important device information—GPS data, messages, call logs, contacts—from an iPhone X and most previous models, according to a NIST test from April. It was able to partially extract data from Twitter, LinkedIn, Instagram, Pinterest, and Snapchat as well. NIST did not test the extraction ability for other apps, like Signal.

UFED 4PC could not extract email data from newer iPhone models, but police can gain access to cloud email services like Gmail with a warrant.

Results from Cellebrite on Android phones

Cellebrite was less successful with phones running Android and other operating systems, though. The UFED tool was unable to properly extract any social media, internet browsing, or GPS data from devices like the Google Pixel 2 and Samsung Galaxy S9 or messages and call logs from the Ellipsis 8 and Galaxy Tab S2 tablets. It got absolutely nothing from Huawei’s P20 Pro phone.

“Some of the newer operating systems are harder to get data from than others. I think a lot of these [phone] companies are just trying to make it harder for law enforcement to get data from these phones ... under the guise of consumer privacy,” Detective Rex Kiser, who conducts digital forensic examinations for the Fort Worth Police Department, told Motherboard. “Right now, we’re getting into iPhones. A year ago we couldn’t get into iPhones, but we could get into all the Androids. Now we can’t get into a lot of the Androids.”

Cellebrite, which did not respond to requests for comment, frequently updates its products to address the failures discovered in testing and in the field, experts said, so the weaknesses NIST identified may no longer exist. Previous NIST testing data, though, shows that many blindspots can last for years.

It is important to note that just because a cracking tool can’t successfully extract data doesn’t mean a forensic investigator can’t eventually get to it. The process just becomes much longer, and requires significant expertise.

Kiser said that Cellebrite is currently the industry leader for most devices. The exception is iPhones, where Grayshift, an Atlanta-based company that counts an ex-Apple security engineer among its top staff, has taken the lead.

Like Cellebrite, Grayshift claims that its GrayKey tool—which it sells to police for between $15,000 and $30,000—can also crack the encryption on any iPhone. And once inside, NIST test results show that GrayKey can completely extract every piece of data off an iPhone X, with the exception of Pinterest data, where the tool achieved partial extraction.

Grayshift did not respond to a request for comment.

Other products, like Virginia-based Paraben’s E3:DS or Swedish MSAB’s XRY displayed weaknesses in acquiring social media, internet browsing, and GPS data for several phones. Some of those tests, though, are older than the recent results for Cellebrite and Grayshift.

In the NIST tests, both Cellebrite and Grayshift devices were able to extract nearly all the data from an iPhone 7—one of the phones used by the Pensacola naval air station shooter. That incident prompted the Department of Justice’s latest call for phone manufacturers to create encryption backdoors, despite ample evidence that hacking tools can break into the latest, most privacy conscious phones, like the iPhone 11 Pro Max.