Security threats evolve as water systems connect to the internet.

By Brett Walton, Circle of Blue

Rye, New York, a wealthy commuter town a 50-minute train ride from central Manhattan, does not seem a typical target for a cyberattack. It is not a major retailer like Home Depot or Target, both of which had millions of credit card accounts stolen in recent years. It is not a military base. It is not home to an important electric plant or substation. Only 16,000 people live there.

Cybersecurity is a huge and emerging public risk.

-Daniel Groves, Arcadis

Yet for three weeks after Labor Day in 2013, the city’s computers were exposed to prying eyes. An Iranian computer expert with links to the country’s spy agencies repeatedly hacked into the system that operates Bowman Dam, a small structure hardly bigger than a barn door that controls water levels on Blind Brook, a calm stream. Marcus Serrano, the city manager, told Circle of Blue that city officials did not know they were being hacked until they received a call from the Department of Homeland Security. The details of the attack are laid out in a Justice Department indictment that was unsealed in federal court on March 24, 2016.

Cybersecurity authorities refer to the Bowman Dam incident as “a read,” meaning that the hacker was looking for information about the dam’s operating system and not attempting to take over the controls. Rye officials do not know why their dam was targeted. They think it may be a case of mistaken identity.

The hacking of Bowman Dam, though not a typical cyberattack, is the most striking example of a problem that U.S. water utilities now acknowledge with greater clarity and vigilance. Computer networks and the industrial control systems that raise dam gates, operate pumps, disinfect drinking water, and guide an assortment of other critical functions are vulnerable to the digital version of breaking and entering. (see sidebar)

In the evolving realm of cybersecurity, where the risks today are different than just a few years ago and the threats tomorrow will be different still, a water utility’s size may not be the most important characteristic for those looking to do harm. “Sometimes cyberattacks are targets of opportunity,” Michael Arceneaux, managing director of WaterISAC, a cybersecurity information hub for the water sector, told Circle of Blue.

The opportunities for malicious activity are increasing. More and more water utilities, in order to save money by remote monitoring, are connecting their control systems to the internet. Meanwhile, hackers are developing computer viruses capable not only of stealing data, but also of taking control of critical infrastructure. “Cybersecurity is a huge and emerging public risk,” Daniel Groves, cybersecurity program manager at Arcadis, a consultancy, told Circle of Blue. “It’s growing more complicated and difficult daily. The attacks are becoming more sophisticated.”

The threats are real, according to cybersecurity experts working for government agencies and in the private sector. Ransomware, the most common cyberattack, according to Arceneaux, locks down computer networks or control systems and demands payment, usually in bitcoins. Though U.S. water utilities have been infected this way, none has paid a ransom, according to Bob Timpany of the Department of Homeland Security. They have recovered their operations by replacing the system or restoring a backed up version.

The Hacking of Bowman Dam Between August 28, 2013, and September 18, 2013, an Iranian computer hacker named Hamid Firoozi broke into the computer system that controls Bowman Dam, in Rye, New York. The details of the attack are laid out in a Justice Department indictment that was unsealed in federal court on March 24, 2016. Firoozi, who is 34, is employed by ITSecTeam, a computer security company based in Iran and sponsored by Iran’s Islamic Revolutionary Guard Corps, one of the country’s spy agencies, according to the Justice Department indictment. Firoozi is also one of seven Iranians involved in a series of cyberattacks against 46 U.S. companies between December 2011 and May 2013, the Justice Department alleges. Rye city officials had no clue that they were being hacked. “We heard about it from the Department of Homeland Security,” Marcus Serrano, the city manager, told Circle of Blue. Federal agents phoned Serrano in September 2013, he recalled, then came to the city offices to pick up the computer and modem that Firoozi had infiltrated. By tapping into the supervisory control and data acquisition system, Firoozi was able to learn water levels and temperatures as well as the status of the sluice gate, which controls the flow of water. Firoozi was not able to operate the gate from Iran because that particular control system had been disconnected for maintenance. “While federal authorities have said these hackers were not able to completely gain control of the dam’s system, this cyberattack surely serves as a bucket of cold water to the face,” said Sen. Charles Schumer (D-NY) on March 24. The Justice Department officially charged Firoozi with unauthorized access of a protected computer. Preet Bharara, U.S. attorney for the Southern District of New York, called the attack “a frightening new frontier in cybercrime.”

Other types of malware are more worrisome threats. Strains such as Havex and BlackEnergy, which was used in an electrical grid attack in December 2015 in Ukraine that shut power to an estimated 700,000 people for a few hours, are designed to take control of critical systems. In a worst case scenario, a water pump station could be overworked to the point that it blows the pipes out of the ground.

The National Infrastructure Advisory Council, a group of experts that advises the Department of Homeland Security and the president on critical infrastructure, says that cybersecurity awareness among water utilities is “often limited” and that the number of cybersecurity experts in the sector is “insufficient for current needs.” Those conclusions are included in a report on water sector resilience to natural disaster and cyberattack that will be released later this month.

For the U.S. water sector, the concern is directed more at the attacks that might occur, rather than the attacks that have already happened.

“To our knowledge, we have not seen anyone manipulate or destroy water infrastructure in the United States” because of a cyberattack, Timpany told Circle of Blue.

It Takes a Village to Rebuff a Cyberattack

Timpany’s official title is the operations chief of the Idaho Falls branch of the Industrial Control Systems Cyber Emergency Response Team, known in security circles as ICS-CERT. It is a division of the Department of Homeland Security.

ICS-CERT is a 9-1-1 switchboard, EMT unit, triage division, and family doctor in one government package. The division, which has a fly-away team for immediate on-site support, specializes in diagnosing computer viruses and rehabilitating infected industrial control systems, or ICS. ICS-CERT calls these systems “the pinnacle of cyber targets” because they have critical importance for the operator, and thus the highest value for the attacker. No mayor or governor desires a citywide blackout or a failure of the sewer system.

Utilities hope that a midnight call to ICS-CERT is never necessary. There is an expanding web of collaborators that are helping to prevent that from happening, or at least minimize the damage if it does. ICS-CERT, in addition to its incident response, conducts vulnerability assessments with utilities that request them. The number of assessments grew from 23 in 2013, to 38 in 2014, to 39 last year.

WaterISAC, another participant, serves as the cybersecurity information center for the water sector. It was formed in 2002, after the September 11 attacks focused attention on critical infrastructure. “We help utilities reduce risk by sending out guidance, checklists, and best practices,” said Arceneaux, the managing director. About 7,500 people who work at water utilities and state agencies are members, he reckoned.

Other groups are working on protocol. The National Institute of Standards and Technology, in response to President Obama’s Executive Order 13636 that was issued in February 2013, developed a cybersecurity framework to provide a common language to talk about cybersecurity risk. The American Water Works Association, the water industry’s main trade groups, is working to achieve widespread adoption of the standards among utility leadership by providing guidance on implementing them.

Do Not Open the Attachment

The basic problem for water utilities today is the convergence of two systems that used to be relatively segregated: information technology (IT) and operational technology (OT). IT is what a layperson commonly associates with cyber threats: the computer systems that are linked to the internet for email, billing, bookkeeping, and desk work. Viruses enter these systems through a mess of pathways: infected USB drives, email attachments, bad links on compromised websites.

Basic cybersecurity starts with the individuals who work at the water utility, says Kevin Morley, security and preparedness program manager with the American Water Works Association.

“The human in the chair is the biggest vulnerability,” Morley told Circle of Blue. “They like to click on things.” Even an e-cigarette charged on a computer’s USB port could be an infection pathway, Morley said.

The OT side is traditionally the engineer’s domain. These are the industrial control systems so critical to drinking water treatment plants, power stations, and other big, centralized infrastructure. They are the targets of Havex, BlackEnergy, and a host of other nasties.

Utilities connect their OT systems to the internet to save time and money on monitoring costs. Remote access means a worker does not have to drive to each pump station to check its performance. Timpany does not tell utilities not to implement wireless connections. But he recommends doing so only if the encryption is secure, passwords are strong, and firewalls are in place between the IT and the OT.

Despite the efforts to prevent cyberattacks on water utilities, Morley argues that the sector needs to be realistic about its capabilities and limitations. Avoiding attack altogether is out of the question, he says.

The human in the chair is the biggest vulnerability. They like to click on things.

-Kevin Morley, American Water Works Association

“If the Pentagon is being hacked by the Chinese and the Iranians, I think we need to have a reality check about a water utility’s ability to stop a state-sponsored cyberattack,” Morley said. He then deadpanned: “A water utility does not have the resources of the Defense Department.” More important is avoiding “cascading consequences” among interconnected water, energy, and public health systems — the loss of a power station that cuts electricity to the water system, for example.

More modeling of potential scenarios could help. Timpany said that ICS-CERT demonstrated the destruction of an electricity generator through cyberattack as a learning exercise, but it has not done that for the water and wastewater sector.

More Awareness But Data Is Scarce

A rigorous accounting of the U.S. water sector’s cybersecurity preparedness is not possible, argues Timpany. Unlike, say, the United Kingdom, which has only 30 water providers, the United States has more than 52,000 public water systems, from big metropolitan networks that serve millions to community services districts that serve hundreds. Only 25 water utilities reported cyber incidents to ICS-CERT in 2015. An unknown number of incidents go unreported, Timpany said.

“It’s hard for anyone to truly articulate the risk in a big, grand statement for the United States water sector because there are so many systems,” Timpany said.

In general terms, however, the trend lines are moving in the right direction.

“The amount of awareness at the management level is much, much higher than it used to be,” Groves, the consultant, said. “I view that as encouraging, that management is starting to understand the risks.”

Morley, too, said that water utilities are showing interest in the topic that wasn’t apparent five years ago.

Yet change comes slowly. The water sector, on the whole, is a conservative industry, Groves said. That is understandable, given the essential service they provide. However, if water systems are going to avoid the digital traps of the 21st century, they need new awareness and new tools: technology to deflect attackers that are constantly probing for weakness; training for managers, particularly at small utilities, who will help integrate the OT and IT worlds; collaboration with vendors who sell the array of control system products; different skill sets for engineers used to dealing primarily with pipes and valves; and an appreciation that cybersecurity will never be solved, only managed.

“It’s an overall cultural change,” Groves explained. “We’re progressing toward it but we have a long way to go.”