At least 100,000 Internet-connected servers sold by Dell, HP, and other large manufacturers contain hardware that is vulnerable to potent remote hack attacks that steal passwords and install malware on their host systems, researchers said.

The threat stems from baseboard management controllers that are embedded onto the motherboards of most servers. Widely known as BMCs, the microcontrollers allow administrators to monitor the physical status of large fleets of servers, including their temperatures, disk and memory performance, and fan speeds. But serious design flaws in the underlying intelligent platform management interface, or IPMI, make BMCs highly susceptible to hacks that can cascade throughout a network, according to a paper presented at this week's Usenix Workshop on Offensive Technologies.

Heightening the risk, a recent Internet scan detected at least 100,000 IPMI-enabled servers running on publicly accessible addresses, despite long-standing admonitions from security professionals never to do so.

"IPMI can be a convenient administrative tool, but under the control of attackers, it can also serve as a powerful backdoor," the scientists from the University of Michigan wrote in the paper, which was titled Illuminating the Security Issues Surrounding Lights-out Server Management. "Attackers who take control of the BMC can use it to attack the host system and network in a variety of ways."

“Parasitic server”

One possibility, the paper continued, is the installation of BMC-resident spyware that captures administrative passwords when an operator remotely accesses a host server. Another scenario: attackers could gain unfettered "root" access to the host by remotely booting the server into recovery mode. Worse yet, attackers could abuse vulnerable BMCs to run an unauthorized operating system on the host that gives raw access to the server disks.

The researchers aren't the first to warn of the threats posed by widely used IPMI and BMC technologies. Last month, Dan Farmer, the highly regarded white-hat hacker, posted his own manifesto that used even stronger language to describe the lurking danger. At one point he wrote:

Imagine trying to secure a computer with a small but powerful parasitic server on its motherboard; a bloodsucking leech that can't be turned off and has no documentation; you can't login, patch, or fix problems on it; server-based defensive, audit, or anti-malware software can't be used for protection; its design is secret, implementation old, and it can fully control the computer's hardware and software; and it shares passwords with a bunch of other important servers, stores them in clear text for attackers to access.

HD Moore, chief research officer of security firm Rapid7 and chief architect of the Metasploit project used by penetration testers and hackers, provides an equally bleak security assessment of IPMI and BMC here.

BMCs contain different names and specifications depending on the server they're bundled with, and there's little public material documenting their inner workings. But because each runs the same IMPI protocol, they're all believed to be susceptible to the same threats. The University of Michigan researchers tested this hypothesis by selecting one such controller, which came embedded on the Super X9SCL-F motherboard of a Supermicro SYS-5017C-LF 1U rack-mounted server. After performing a thorough analysis of the device, the scientists found that its firmware (designed by a firm called ATEN Technology) contained "numerous textbook security flaws, including exploitable privilege escalation, shell injection, and buffer overflow vulnerabilities." The researchers developed proof-of-concept attack code that exploited the vulnerabilities to remotely obtain root access on the BMC. (Supermicro has since issued BMC firmware updates that fix some or all of the vulnerabilities.)

They went on to catalog a list of attack scenarios malicious hackers could mount when exploiting the bugs. They included:

Subverting the host system or other machines on the management network

Installing BMC spyware that eavesdrops on remote management sessions to sniff passwords or even the physical server console

Installing persistent BMC rootkits that provide attackers with backdoor access that remains hidden from IPMI logs

The creation of IPMI botnets to take advantage of the large amount of network bandwidth at their disposal

In all, the scientists detected more than 100,000 Internet-exposed IMPI devices, 40,000 of which used the Supermicro BMC they tested at length.

"We conservatively estimate that it would take less than an hour to launch successful parallel attacks against all of the 40,000 ATEN-based Supermicro IPMI devices that we observed listening on public IP addresses," they reported.

Either incompetence or indifference

The paper includes a list of defenses that should be required reading for anyone who administers a server anywhere. Suggestions include keeping IPMI firmware up to date, changing default passwords, and never, ever running IPMI devices on public IP addresses. This last admonition is widely repeated—often by the manufacturers of the servers that are put at risk by the vulnerabilities. The scientists' Internet scans provide convincing evidence that this advice is frequently ignored, so unfortunately, it's worth repeating often.

But the researchers also take engineers at original equipment manufacturers (OEMs) to task for, among other things, building devices that have IPMI capabilities turned on by default. The researchers go on to direct some harsh words at the people developing IPMI devices and the servers they go into.

"Given the power that IPMI provides, the blatant textbook vulnerabilities we found in a widely used implementation suggest either incompetence or indifference towards customers' security," the paper states. "While some OEMs recommend helpful precautions such as dedicated management networks, this should not be an excuse to shift blame to users who fail to heed this advice and suffer damage because of vulnerabilities in IPMI firmware. We believe that properly securing IPMI will require OEMs to take a defense-in-depth approach that combines hardening the implementations with encouraging users to properly isolate devices."