A survey of more than 6,000 firmware images spanning more than a decade finds no improvement in firmware security and lax security standards for the software running connected devices by Linksys, Netgear and other major vendors.*

When networking hardware giant NETGEAR issues a security advisory for one of its many products, the company is careful to include a bit of boilerplate text at the end, acknowledging the work of independent security researchers who often bring serious software security issues to the company’s attention.

“NETGEAR constantly monitors for both known and unknown threats. Being pro-active rather than re-active to emerging security issues is fundamental for product support at NETGEAR.”

Home networking stalwart Asus makes similar claims on its web page.

“We take every care to ensure that ASUS products are secure in order to protect the privacy of our valued customers. We constantly strive to improve our safeguards for security and personal information in accordance with all applicable laws and regulations,” the company claims.

Those are comforting words. But an extensive study of thousands of device firmware images by NETGEAR, ASUS and 16 other vendors suggests that there is little truth behind them; the security of device firmware is terrible and has not improved in any measurable way over the last 15 years, even as attacks on connected devices like home routers have spiked.

“Nobody is trying.”

“Nobody is trying,” said Sarah Zatko, the Chief Scientist at the Cyber Independent Testing Lab (CITL), a non-profit organization that conducts independent tests of software security. “We found no consistency in a vendor or product line doing better or showing improvement. There was no evidence that anybody is making a concerted effort to address the safety hygiene of their products,” she said.

Zatko presented the findings of CITL’s extensive study in Las Vegas on Friday on the sidelines of the Black Hat and DEF CON conferences at an event hosted by The Hewlett Foundation. CITL was started by Sarah and her husband Peiter (aka “Mudge”) Zatko. It bills itself as a kind of “Consumer Reports” for cyber security, partnering with that organization as well as The Ford Foundation, The Digital Standard and online payments firm Stripe.

Sarah Zatko is the Chief Scientist at CITL

The CITL study surveyed firmware from 18 vendors including ASUS, D-link, Linksys, NETGEAR, Ubiquiti and others. In all, more than 6,000 firmware versions were analyzed, totaling close to 3 million binaries created from 2003 to 2018. It is the first longitudinal study of IoT software safety, according to Zatko.

CITL researchers studied publicly available firmware images and evaluated them for the presence of standard security features such as the use of non-executable stacks, Address Space Layout Randomization (ASLR) and stack guards, which prevent buffer overflow attacks.

Security Blind Spots in Firmware

The results were not encouraging. Time and again, firmware from commonly used manufacturers failed to implement basic security features even when researchers studied the most recent versions of the firmware.

For example: firmware for the ASUS RT-AC55U wifi router did not employ ASLR or stack guards to protect against buffer overflow attacks. Nor did it employ a non-executable stack to protect against “stack smashing,” another variety of overflow attack.

CITL found the same was true of firmware for Ubiquiti’s UAP AC PRO wireless access points, as well as DLink’s DWL-6600 access point. Router firmware by vendors like Linksys and NETGEAR performed only slightly better on CITL’s assessment.

There were some bright spots. Almost all recent router firmware by Linksys and NETGEAR tested by CITL boasted non-executable stacks. However, those same firmware binaries did not employ other common security features like ASLR or stack guards, or did so only rarely, CITL data suggest.

Zatko said the CITL testing is by no means comprehensive. Just the opposite. In order to test so many firmware versions, CITL had to focus on security features that could be verified “at scale.” That ruled out checks that require more hands-on assessments, such as looking for hardcoded passwords, data and privacy leaks or exploitable vulnerabilities.

CITL data on major vendor firmware images from 2018 show no evidence of progress on security measures. (Data courtesy of CITL.)

“Stack guards and buffer overflow protection are the canaries in the coal mine,” she said: basic protections that all software should employ. The absence of even basic protections suggests that the tested firmware may contain more serious vulnerabilities and that firmware security is years behind the security of applications like Windows, OS X or Google Chrome and FireFox.

“These are the seatbelts and airbags of the software world. These numbers are unheard of in operating systems or (Web) browsers. Its just a sign that they’re not trying,” Zatko said.

Even worse, CITL researchers found no clear progress in any protection category over time, said Zatko. Researchers documented 299 positive changes in firmware security scores over the 15 years covered by the study…but 370 negative changes over the same period. Looking across its entire data set, in fact, firmware security actually appeared to get worse over time, not better, CITL said.

CITL assessments of router firmware from 2003 through 2018 show no consistent improvement in security.

Looked at longitudinally, the results of the CITL testing show no clear trend. “It’s pretty random looking,” Zatko told Security Ledger. In fact, security scores for the same vendor diverge over time rather than converge as the number of product lines the company supports increase, Zatko observed.

In an email statement, Linksys Vice President of Communications Jen Warren said that the company tests binaries for known security vulnerabilities and that “the security features described add no value if you have physical access to the router –these features help prevent exploitation of an application vulnerability that is exploitable over the network.”

Other vendors including ASUS and Ubiquiti did not respond to emails from Security Ledger seeking comment on the CITL research.

A Failure with many Fathers

CITL doesn’t point the finger at any single cause for the lack of security in firmware. Zatko said the persistent failure, across so many vendors, is the product of systemic failures in how software is produced.

“It’s a bottom up problem,” Zatko told Security Ledger in an interview. “Educators in (computer science) curricula don’t talk about this. Organizations that make software don’t talk to developers about it. There are no organizational rules about ‘You have to use these build flags’ or ‘You have to do this post build testing.'”

In many cases, enabling security features is as simple as enabling a small set of options during the software “build,” when firmware source code is turned into executable “binaries” that are shipped with routers, cameras, smart television sets and other hardware.

“It’s like every stage in the chain had a blind spot about these basic safety features that have been around forever so they took them for granted and assumed they’re there,” she said.

The poor scores suggest that many companies making devices for the Internet of Things have not adapted their practices to account for the increased security risks that accompany connected devices.

“These are companies that were primarily in hardware until recently,” Zatko said. “Mostly they’re doing the wrong thing.”

Mounting Attacks on Embedded Devices

The stakes are high. Attacks that target routers, cameras and other non-personal computer devices are on the rise. In 2016, the Mirai botnet, which was made up largely of compromised digital video recorders and Internet connected cameras, showed the power that tens of thousands of small, compromised devices could have.

Zatko said attackers are turning to those devices not for a challenge, but because they are easy prey, with software protections that are often years behind those in Windows, Apples OS X or on platforms like Google Chrome.

Despite that, Zatko said much of the conversation about IoT security focuses on software applications, but misses the security of firmware. Hardware vendors need to do much more to harden their devices prior to release. That means assessing the security of software development tools and implementing basic hardening consistently.

Such steps are easy to implement, will pay big dividends and are measurable, she said. There is no evidence any of the vendors studied by CITL plans to implement them.

(*) Updated to add comment from Linksys – PFR August 14 2019