A pair of researchers have uncovered more than two dozen vulnerabilities in products used in critical infrastructure systems that would allow attackers to crash or hijack the servers controlling electric substations and water systems.

The vulnerabilities include some that would allow an attacker to crash or send a master server into an infinite loop, preventing operators from monitoring or controlling operations. Others would allow remote code-injection into a server, providing an opportunity for an attacker to open and close breakers at substations and cause power outages.

"Every substation is controlled by the master, which is controlled by the operator," says researcher Chris Sistrunk who, along with Adam Crain, found vulnerabilities in the products of more than 20 vendors. "If you have control of the master, you have control of the whole system, and you can turn on and off power at will."

The vulnerabilities are found in devices that are used for serial and network communications between servers and substations. These products have been largely overlooked as hacking risks because the security of power systems has focused only on IP communication, and hasn't considered serial communication an important or viable attack vector, Crain says. But the researchers say that breaching a power system through serial communication devices can actually be easier than attacking through the IP network since it doesn't require bypassing layers of firewalls.

An intruder could exploit the vulnerabilities by gaining physical access to a substation – which generally are secured only with a fence and a webcam or motion-detection sensors – or by breaching the wireless radio network over which the communication passes to the server.

"If someone tries to breach the control center through the internet, they have to bypass layers of firewalls," Crain said. "But someone could go out to a remote substation that has very little physical security and get on the network and take out hundreds of substations potentially. And they don't necessarily have to get into the substation either."

He points to a recent presentation at the Black Hat security conference that discussed methods for hacking wireless radio networks, which a lot of utility control systems use, including ways to crack the encryption.

"There are quite a few ways onto these networks, and utilities have to worry about this new attack vector," Crain said.

Once in the network, an intruder can send a malformed message to the server to exploit the weakness.

"The device is supposed to throw that [malformed] message away," says Sistrunk, "and in these cases it's not and is causing issues."

Neither Crain nor Sistrunk is a security researcher. Sistrunk is an electrical engineer at a major utility, but conducted the research independently of his employer and therefore asked that it not be identified. Crain recently launched a consulting firm called Automatak that focuses on industrial control systems. They began to examine the systems last April using a fuzzer that Crain created, and submitted their findings to the Department of Homeland Security's Industrial Control System-CERT, which helped them notify the vendors.

"We found vulnerabilities in virtually all implementations [of the protocol]," Sistrunk said. "Some of them are worse than others."

Since then, ICS-CERT has published a number of advisories about the vulnerabilities, and vendors have distributed patches for nine of them, but the rest remain unpatched so far. Despite the distribution of patches, Crain and Sistrunk say that many utilities have not applied them because they're unaware of the serious nature of the vulnerabilities.

The systems use DNP3, a protocol for serial communications that is used in almost all electrical utilities in the U.S. and Canada to transmit communication between servers located in data centers and field devices. Electric utilities generally have a data center with two or three servers that can each monitor and communicate with a hundred or more substations, depending on the size of the utility.

The servers communicate with programmable logic controllers and remote-terminal units in the field to collect status data from them in order to allow operators to monitor conditions and to allow them to trip breakers as needed or to increase or decrease the voltage.

Causing the server to crash or enter an infinite loop would blind operators to conditions in the field – something they might not initially realize since a crashed server in the data center doesn't always register to operators, who work in other locations. Sistrunk says it would likely take operators a while to notice that the data they're seeing on their screens, which is fed by the servers, hasn't refreshed in a while. In the meantime, they might make bad decisions based on outdated data.

A lot of utilities also use the master servers for security purposes to control alarm systems, so crashing them would potentially disable alarms as well.

Sistrunk says a reboot of the server will generally resolve the issue, but an intruder could continue to send malicious messages to the server causing it to crash repeatedly. He also said that in some cases they found that the attack would corrupt the system configuration, which meant the system had to be reconfigured or restored from a backup before operations returned to normal.

Of the 25 vulnerabilities they uncovered the most serious was the buffer overrun vulnerability that would allow someone to inject arbitrary code into the system and own the server.

One of the vulnerabilities they found exists in the source code for a popular library from Triangle Microworks. It's not known how many vendors and products have used the library and are therefore vulnerable, but Crain and Sistrunk say that the library is one of the most popular among vendors and is used by 60 to 70 percent of them for their products.

Crain says the standard for DNP3 is not the problem but that the vulnerabilities are introduced in the insecure ways that vendors have implemented it.

The problem is exacerbated by the fact that separate security standards set by the North American Electric Reliability Corporation for how to secure power systems focus only on IP communications, overlooking the real vulnerabilities that serial communications also present.

The researchers plan to discuss their findings at the S4 security conference to be held in Florida in January.