In his years-long career developing software for power grids, Stan McHann had never before heard the ominous noise that rang out last Wednesday. Standing in the middle of a utility command center, he flinched as a cyberattack tripped the breakers in all seven of the grid's low voltage substations, plunging the system into darkness. "I heard all the substations trip off and it was just like bam bam bam bam bam bam bam bam," McHann says. "The power’s out. All you can do is say, OK, we have to start from scratch bringing the power back up. You just take a deep breath and dig in."

Thankfully, what McHann experienced wasn't the first-ever blackout caused by a cyberattack in the United States. Instead, it was part of a live, week-long federal research exercise in which more than 100 grid and cybersecurity experts worked to restore power to an isolated, custom-built test grid.

In doing so they faced not just blackout conditions and rough weather, but also a group of fellow researchers throwing a steady barrage of cyberattacks their way, hoping to stymie their progress just as a real enemy might.

Power Play

Funded by the Defense Advanced Research Projects Agency, the exercise, which ran the first week of November, served as a testing scenario for seven DARPA-developed grid recovery tools.

But while the situation was manufactured, the conditions of the exercise were all too real. Researchers built their test grid off of the already isolated power grid on Plum Island, a Department of Homeland Security animal disease research facility at the tip of Long Island's North Fork. Roughly the size of Manhattan's Central Park, Plum Island sits about three miles offshore in the Long Island Sound, and is accessible only by ferry. In addition to DHS's livestock research facility, Plum Island is also home to ruins from armaments and fortresses built during World War I and II, pristine beaches, a lighthouse built in 1898, and even packs of gregarious harbor seals in the winter.

The result: A surreal combination of utilitarian federal operations, breathtaking natural habitat, untapped Hamptons real estate, and a nagging sense of foreboding. (Despite persistent conspiracy theories, DHS representatives patiently but firmly deny that there is anything creepy about the island.)

"When we first started the program, we were working in university labs and simulating everything," says Walter Weiss, the DARPA program manager who oversees the agency's research into restoring power to a dead grid—what utilities call "black start."

During one early RADICS meeting, Weiss convinced the host university to cut power to the floor the team was on, forcing researchers to consider how the tools they were developing would remain effective during a blackout. "We said, 'Imagine you're going to an island,'" Weiss says, laughing.

Assume Breach

Over the past few years, the threat of grid hacking has morphed from a distant possibility to a stark reality. The most chilling incidents to date are two cyberattack-induced blackouts in Ukraine—one in December 2015 and the next a year later in December 2016—that caused power outages for hundreds of thousands of residents in Kiev for a few hours each time. Both attacks are thought to have been perpetrated by Russian state-sponsored hackers. And though a similar incident hasn't played out in the US so far, there is increasing evidence that various hacker groups have infiltrated US grid defenses. The Department of Homeland Security warned repeatedly this year that it has detected extensive Russian probing of the US grid.

But awareness can only get you so far. For actual resilience, the industry needs what cybersecurity practitioners call an "assume breach" mentality: thinking not just about how to keep attackers out, but knowing how to respond if and when they do break in.

"When we first started the program, we were working in university labs and simulating everything." Walter Weiss, DARPA

Since the end of 2015, DARPA's Rapid Attack Detection, Isolation and Characterization Systems program, which Weiss oversees, has taken up that mantel for power grids. RADICS seeks to develop tools that aid in three phases of black start after a cyberattack. The first involves creating sensors that can give accurate readings and situational awareness even after a hack has potentially skewed or degraded the reliability of existing monitoring equipment. The second looks at developing specialized equipment for rapidly setting up a secure backup network in a pinch, since whatever malware caused the blackout may still infect some systems. And the third focuses on tools that can quickly scan for threats to help understand how an attack happened, and how to lock down any remaining hacker footholds as power comes back online.