The last couple of weeks have been a fascinating roller-coaster in the land of embedded device security. After the Mirai source code leaked earlier this year, I outlined the idea that "smart" botnet operators would quickly adapt to use new methods of spreading, beyond just telnet scanning, in order to quickly gain "dominance" and build larger botnets. As predicted, this has happened.

Now, this post is NOT going to be about Mirai, as quite frankly, everyone else has written quite enough already on that subject, but instead, will be about the systemic failures in how device vendors implemented the TR-064 protocol that lead to this situation in the first place.

So, firstly, what even is TR-064?

TR-064 is the DSL Forum specification for "LAN Side DSL CPE Configuration". This specification outlines a protocol via which a CPE device (i.e. router/modem) can be configured from the LAN side, via software on a PC on said LAN. You may have encountered TR-064 if, for example, your ISP provided you with an "Internet Setup" CD-ROM or software package with your router.

The specification outlines a SOAP based protocol similar to UPnP, via which devices configurations can be viewed and altered, allowing setup of routers. Via this protocol, you can do things like change the DNS servers, NTP servers, WiFi settings, ACS server settings, and suchlike.

TR-064 Security Model (as per specification).

The specification document addresses security early on - in section 4 - wherin it states that "Access to any action that allows configuration changes to the CPE MUST be password protected.". For authentication, the use of HTTP Digest based auth (HTTP Basic Auth, effectively) is referred to.

Furthermore, it suggests that SSL/TLS may be used, and that "Sensitive information, such as passwords, MUST NOT be readable at all.". It also mentions that for "Read Only" operations, such as status/statistics, authentication "SHOULD NOT" be required.

The document does not specifically state that the service should not be accessible via the WAN interface, however, it does mention that the specification is not intended for WAN side management of devices.

Catastrophic Implementation Failure.

As explained above, the specification states the following things.

Read-Only calls do not require authentication. Calls that "write", or alter device configuration, MUST require authentication. Passwords MUST NOT be readable via this protocol.

The reality of the situation is far, far different. Vendors either didn't bother read the specification properly, and thus missed the "security" section, or overlooked its importance. Or just decided "fuck it, too much effort" and sacked it off, as these criteria are in no way adhered to by the implementation which is found in the real world.

In reality, a large number of vendors have shipped a TR-064 implementation (as part of the RomPager CWMP Server) that requires NO AUTHENTICATION WHATSOEVER, is available on ALL interfaces (i.e. - is available on the WAN interface), and does ABSOLUTELY NOTHING to prevent passwords from being retrieved.

This is, in my opinion, a criminally negligent level of systemic failure to not only properly adhere to the specification, but also, to take the security of the devices they are shipping into consideration.

The Impact of Exposing Unauthenticated TR-064 Interfaces to the Internet.

The actual impact of exposing a TR-064 interface to the internet without any security controls or authentication has been widely understated by both the security community, and the press, potentially due to a lack of understanding of the severity of the issue, and also due to being distracted by a secondary vulnerability being exploited actively by the Mirai botnet leading to internet outages across the world.

While the vulnerability being exploited by Mirai (which I will get to later) is "very bad", it is distracting people from what I feel is the real impact of the issue, which I hope to outline below.

So, the "real impact" of this issue (exposed, unauthenticated, TR-064 interfaces on CPE devices), is effectively that an attacker can completely reconfigure the device with no authentication whatsoever.

This would allow an attacker to do a whole host of things, limited only by their "imagination" and ability to comprehend a fairly simple specification document. Below I will outline some example attacks that could be conducted, which are by no means an exhaustive list, just some trivial ones that came to mind, all of which have severe implications.

Creating new firewall rules/port mappings (giving access to internal networks... Or telnet/SSH/admin interface on device!). This attack is fairly simple to pull off - the "urn:dslforum-org:service:WANIPConnection:1" interface seems to have the ability to create new "port forwards" in the way UPnP does, which would allow forwarding devices on the "internal" (LAN Side) to the "external" (WAN side) of the device. This enables a remote attacker to access hosts that would not normally be accessible, enabling further attacks. Recovering the WPA Passphrase, wireless MAC address, wireless SSID. (the WPA Passphrase is usually the admin password! If you are to enable access to the web interface, or SSH/Telnet interface, via the above method, you can conduct further attacks on the device using the recovered credentials. Or simply build a massive database of stolen wireless network credentials for future abuse.) Change the DNS servers on the devices so one can do pharming attacks. By reconfiguring the devices DNS servers, you can force the people using the affected device to use you as their DNS provider. This is a trivial feat to accomplish, and by doing so, you can redirect their internet traffic to phishing sites, or sites serving up malware to infect their devices. Pharming attacks on routers are not new, and are highly effective ways to steal data. All one would need to do is set up a malicious DNS server, and some phishing sites. Change the ACS Provisioning Servers on the devices. This means the ISP can't fix them anymore and one will have total control over them. Setting up your own ACS server is not particularly difficult. If an attacker reconfigures the ACS settings on the device, they not only are able to upload their own firmware images to the device (say, for example, backdoored firmware created using RPEF), but they also lock the ISP's engineers out of the device entirely, rendering it impossible for them to fix the device without either shipping a replacement to the customer, or sending an engineer around to fix it. Sending a replacement would probably involve an engineers visit in a lot of cases. Even worse - if an attacker was to install backdoored firmware on the device, the device will remain infected even after a reboot. Attackers have deployed malicious firmware images in the past as a persistence mechanism, as I discovered in 2013 when Linux\Flasher.A happened.

The SetNTPServers Command Injection Vulnerability, and Mirai.

Just to make matters worse (or, more hilarious), a trivially exploited command injection vulnerability exists in the vulnerable TR-064 implementation, as disclosed by kenzo2017. If you send specially crafted input to the "SetNTPServers" SOAP endpoint, with a command wrapped in backticks (like: `id`) in the "NewNTPServer1" (or "NewNTPServer2"... etc) field, the command will be executed with root (admin) privileges on the device.

This vulnerability is being widely exploited in the wild by a variant of the Mirai botnet, to infect devices in a worm-like fashion. The bot, when installed, disables access to the TR-064 (and TR-069) port on the device, effectively preventing reinfection, and also locking out ISP engineers from the device. While a reboot will disinfect the device, it remains vulnerable, and will probably be reinfected within minutes.

The virulent spread of this worm has had catastrophic consequences for ISP's globally, with outages being widespread as customers routers cease functioning properly. Affected ISP's include Deutsche Telekom, KCom, TalkTalk, Post Office Broadband, Eir, and many, many others. We have observed vulnerable devices on IP ranges associated with Demon Internet, Plusnet, Vodafone Ireland, VEVO in Brazil, and many others.

When we ran some (limited, non intrusive) scans of a small portion of the internet, we discovered over 500,000 vulnerable devices. Given that Deutsche Telekom alone had almost a million vulnerable devices on their network, the scale of this issue is absolutely enormous, with potentially over 10 million devices globally being vulnerable to the issue, as an estimate based on the data we had.

The Patching Race Condition, or, why fixing this is going to be a fucking nightmare.

So, as mentioned above, the current attacks disable both TR-064 and TR-069 access to the infected devices. This prevents both reinfection, and engineers from the ISP from accessing the devices to update the software on them. While a reboot of the device will remove the infection, the device will be reinfected within minutes.

This leads to a classic "Race Condition" issue for ISP's attempting to roll out fixes to affected customers. When a customers device is rebooted, how quickly can the ISP push a patch to the device? Will the device be patched before it is reinfected? Thus far, based on our observations, we believe that currently, the worm is "beating" the ISP to the punch in many cases, severely slowing down the rate at which fixes can be rolled out to devices, and further prolonging the outages for customers.

ISP's could engage in network-level filtering (blocking access to the ports from external ranges), however, despite their claims of doing so, we have not really seen it have much effect. This points to probable deficiencies in the ISP's abilities to properly filter traffic between their endpoint devices.

It could be worse, if the attackers were smarter...

With the current state of affairs, the attackers are not being particularly clever in how they are exploiting the issue. Their method of exploitation trashes the NTP configuration on the devices, and causes the devices to cease functioning reliably, making it immediately obvious to impacted users that something is deeply wrong.

Furthermore, their malware is not persistent, and it does not permanently lock the ISP out of the device, so patches can be rolled out (with the caveat that they have to "win" a race against the worm itself...). It also does not attempt to redirect users internet traffic for phishing, or capture credentials.

Had the attackers done more testing, and more research into the TR-064 protocol before running wild with their mass-exploitation, they could have created a situation in which the affected ISP's would have had no way to fix the devices beyond shipping replacement units to all of their affected customers - an endeavor that would cost millions of pounds (what is the cost-price of a ZyXEL or D-Link router? Can they be obtained in bulk on short notice?) in device costs, shipping costs, and engineer visits to peoples homes to set up the devices.

Such a thing would have been completely catastrophic, and we should be somewhat thankful that we are dealing with script kiddies here, and not more advanced attackers!

Some Conclusions.

In conclusion, the state of affairs is pretty terrible. What we have here, is a series of systemic failures by device vendors to secure their devices which has lead to global internet outages for millions of people, probably lots of financial losses for businesses and end users, along with stress for end users affected - and network engineers trying to mitigate the issue.

While the situation could be far worse, as it stands, there are some important lessons we could learn from not only this, but from the prior (and ongoing) Mirai epidemic.

Vendors should be held liable for neglecting to properly consider the security of their products when they ship them. A flaw such as this constitutes, in my view, blatant criminal negligence on their behalf, and there should be a legal framework in place to hold them responsible for damages caused. ISP's should "up their game" when it comes to having the ability to respond to incidents such as this, better abilities to engage in filtering of attacks and blocking of traffic would have helped greatly in slowing down the spread of this issue. ISP's, and device vendors, should also probably engage with the security industry to audit their products prior to shipping them, so that these issues can be caught early on and never make it past the door. People implementing things should read the bloody specification before implementing it badly, and do gap-analysis of some form to ensure that what they have created actually adheres to the specifications fully, especially when it comes to security implications. This will not be the last time we see an issue like this. Attackers are more than ready to quickly adapt their exploitation tools to leverage new vulnerabilities, especially ones as widespread as this. Vendors and ISP's need to react faster, and roll out patches quicker, in the future.

Below we have a list of impacted devices. I will update this list or provide a link to an updated list if we find more devices/vendors who are vulnerable to the issue.

Impacted Vendors (this list is probably not complete):

Arcadyan

Aztech

BEC

Billion

Binatone

BiPAC

BTTL

Comtrend

Digicom

D-Link

DT

Gateway

GMTK

iBall

ITI

MitraStar

QTECH

Supernet

T-Com.

Teracom

T-Home

TP-LINK

Upvel

ZTE

ZyXEL

Impacted Devices:

Arcadyan ADSL Router

Aztech DSL5001

Aztech DSL5005

Aztech DSL5008

Aztech DSL5018EN

Aztech DSL5068EN(1T1R)

Aztech RAW300-USB

Aztech Technologies Pte. Ltd. 700WR

Aztech Technologies Pte. Ltd. DSL5028EN(1T1R)

Aztech Technologies Pte. Ltd. DSL5028EN(2T2R)

Aztech Technologies Pte. Ltd. RAW300L-A05

BEC 6300NEL R10

BEC 6300VNL

BEC 6300VNL R4

BEC 6300VNL R5

BEC 6300VNL R6

BEC 9800VN

Billion BiPAC 4500VNOZ

Billion BiPAC 6300NX

Billion BiPAC 6300NXL

Billion BiPAC 6801VNL

Billion BiPAC 7300WR2

Billion BiPAC 8300NL

Billion BiPAC 8400NLR2

Billion BiPAC 8400NL-T R2

Binatone Binatone Router Adsl Router

Binatone DT820

Binatone DT 820

Binatone DT845W

Binatone DT850W

Binatone DT850W DSL Gateway

Binatone DT850W(HFCL)

Binatone DT860W

BiPAC 8300NL

BTTL 110TC2

BTTL 450TC1

BTTL 450TC2

Comtrend VR-3041u

Digicom RAW150-A02

Digicom RAW150-A03

Digicom RAW300-A01

Digicom RAW300U-A02

D-Link DSL-2875AL

D-Link DSL-2877AL

D-Link DSL-3780

D-Link DSL-3782

D-Link DSL-3882

DT Speedport Entry SPW505V

Gateway Prestige 660HNU-T1

GMTK WSTH-136GN

iBall iB-WRA150N DSL-Gateway

ITI Ltd. DNA-1051

ITI Ltd. DNA-2012

ITI Ltd. DNA-2013

MitraStar DSL-100HN-T1-GV

MitraStar DSL-100HN-T1-NV

MitraStar DSL-100HN-T1v4

MitraStar DSL-2401HN-T1C

MitraStar DSL-2401HN-T1C-GV

MitraStar DSL-2401HN-T1C-NV

MitraStar HGW-2501GNP-NV

MitraStar HGW-2501GN-R2 QTECH QTECH ROUTER

Supernet DSLW200

Supernet DSLW200 ADSL2+W2

T-Com. Speedport W303V

T-Com. Speedport W303V DTW303VA

T-Com. Speedport W 303V Typ A DTW303VA

T-Com. Speedport W 502V Typ A DTW502A

T-Com. Speedport W 503V Typ C DTW503VA

T-Com. Speedport W700V DTW7V

T-Com. Speedport W720V DTW72V

T-Com. Speedport W722V DTW722V

Teracom TDSL300W2 ADSL2+W2

T-Home Speedport W 504V Typ A DTW504VA

T-Home Speedport W 723V Typ B DTW723VA

T-Home Speedport W 921V DTW921V

TP-LINK TD-W8951ND DSL-Gateway

Upvel LLC. 354AN4G

Upvel LLC. UR-314AN4G

Upvel LLC. UR-354AN4G

ZTE ZXHN H108N

ZTE ZXV10 W300

ZTE ZXV10 W300B

ZTE ZXV10 W300S

ZyXEL AMG1001-T10A

ZyXEL AMG1201-T10A

ZyXEL AMG1202-T10A

ZyXEL AMG1202-T10B

ZyXELAMG1202-T10B

ZyXEL AMG1302-T10A

ZyXEL AMG1302-T10B

ZyXEL AMG1302-T11B

ZyXEL AMG1302-T11C

ZyXEL AMG1312-T10B

ZyXEL DEL1201-T10A

ZyXEL DEL1202-T10B/B

ZyXEL DEL1202-T10B/W

ZyXEL DEL1312-T10B

ZyXEL eircom D1000 Modem P-660HNU-T1 v2

ZyXEL eir D1000 Modem P-660HNU-T1 v2

ZyXEL P-1202-T10B

ZyXEL P-1302-T10B

ZyXEL P-1302-T10D

ZyXEL P-1302-T10D v2

ZyXEL P-1302-T10D v3

ZyXEL P660HN Lite EE

ZyXEL P-660HN-T1A

ZyXEL P-660HN-T1A_IPv6

ZyXEL P-660HN-T1A v2

ZyXEL P-660HN-T1H

ZyXEL P-660HN-T1H_IPv6

ZyXEL P-660HN-T1_IPv6

ZyXEL P-660HN-T1 v2

ZyXEL P-660HN-T3A

ZyXEL P-660HN-T3A_IPv6

ZyXEL P-660HNU-T1

ZyXEL P-660HNU-T1_IPv6

ZyXEL P-660HNU-T1 Prestige 660HNU-T1

ZyXEL P-660HNU-T3_IPv6

ZyXEL P-660H-T1 Prestige 660H-T1

ZyXEL P-660H-T1v3s

ZyXEL P660HT3 EE

ZyXEL P660HTN EE

ZyXEL P-660HU-T1 Prestige 660HU-T1

ZyXEL P-660N-T1A

ZyXEL P-660R-T1

ZyXEL P-660R-T1 v3

ZyXEL P-660R-T1v3s

ZyXEL P-660R-T1 v3s

ZyXEL P-660R-T3-v3s



