Nation-sponsored hackers have penetrated the operational networks multiple US and European energy companies use to control key parts of the power grid that supplies electricity to hundreds of millions of people, researchers warned Wednesday.

The incursions detected by security firm Symantec represent a dramatic escalation by a hacking group dubbed Dragonfly, which has been waging attacks against US and European energy companies since at least 2011. In 2014, Symantec reported that Dragonfly was aggressively establishing beachheads in a limited number of target networks, mainly by stealing the user names and passwords used to restrict access to legitimate personnel. Over the past year, the hacking group has managed to compromise dozens of energy firms and, in a handful of cases, install backdoors in the highly sensitive networks the firms use to supply power to the grid.

"What's most concerning is we now see them intruding on operational networks of energy companies," Eric Chien, technical director of Symantec's security response and technology division, told Ars. "Before, we were talking about them being one step away, and what we see now is that they are potentially in those networks and are zero steps away. There are no more technical hurdles for them to jump over."

The escalation is troubling because operational networks—sometimes called electronic security perimeters in the energy industry—can often wield significant influence over the stability of the electric grid they're responsible for. In the Northeast Blackout of 2003, a contributing cause was the failure of a system in an operational network that tracked the health of the grid in real time. When a separate fault occurred, the grid supplying electricity to 55 million people shut down.

At a minimum, attackers who have control of a company's operational network could use it to become de facto operators of the company's energy assets. That control includes the ability to turn on or off breakers inside the companies' infrastructure and hijack systems that monitor the health of the grid. That's an unsettling scenario, but there's a more troubling one still: the attackers might also be able to use their control of multiple grid-connected operational networks to create the kinds of failures that led to the Northeast Blackout of 2003. Chien said Symantec has recently issued private warnings to more than 100 energy companies and organizations, including the North American Electricity Reliability Corporation and the US Department of Homeland Security. On Wednesday, it was expected to publish a public warning here.

The Symantec report stressed that simply removing malware from infected networks wasn't enough to counter the threat because in many cases the attackers have the credentials and other data needed to regain control. Wednesday's report provides a variety indicators energy companies can use to tell if their networks have been compromised by Dragonfly. It also lists several best practices for avoiding future compromises, including the use of long, randomly generated passwords that can't be guessed when attackers get ahold of the corresponding cryptographic hash.

Wouldn't be the first time

If Symantec's worst fears were to materialize, it wouldn't be unprecedented. In December 2015, a hack attack on a power distribution center just outside Kiev, the capital of Ukraine, caused about 225,000 people to lose power for as long as six hours. It was the world's first known instance of someone using hacking to generate a real-world power outage. Almost to the day one year later, a hack attack on a Ukrainian power transmission facility caused a smaller number of Kiev residents to lose power for about an hour. Researchers have attributed the attacks to a hacking group dubbed Sandworm.

In the 2015 attack, Sandworm used a revamped version of a tool known as BlackEnergy to break into the corporate network of the targeted power companies and from there to collect passwords and other data that would allow the hackers to penetrate the supervisory control and data acquisition systems the companies used to generate and transmit electricity. Sandworm then used the access to open circuit breakers that cut power. In 2016, Sandworm was back with a new piece of malware dubbed Crash Override by some researchers and Industroyer by others. The custom malware was designed specifically to attack electric grid systems by using the same arcane technical protocols that individual systems rely on to communicate with one another.

Dragonfly, by contrast, uses a completely different set of tools, leading Chien to believe the two groups are completely separate. Both the earlier Dragonfly campaigns in 2013 and 2014 and the group's more recent attacks relied solely on backdoors and remote access trojans. From there, the attackers might use their access to operational networks to manually control the breakers in much the way Sandworm did in the 2015 attack. It's also possible Dragonfly might deploy an as-yet unseen piece of malware that automates malicious functions similarly to how Crash Override did.

After this Ars post went live, several security professionals with expertise in electric grids downplayed the likelihood of the operational network compromises being used to cause blackouts or take down parts of the grid. Robert Lee, the founder and CEO of Dragos Security, said the hackers would need more than the mere ability to control human machine interfaces that flip switches and open and close breakers. While he said an attack that mimicked the techniques that disrupted Ukrainian power in 2015 was possible, he said differences in the US grid would make those tactics much less effective. Lee's Twitter thread below is well worth reading all the way through:

Lots of buzz about Symantec's Dragonfly 2.0 so I'd like to add some first impression thoughts in this thread. (1/X) — Robert M. Lee (@RobertMLee) September 6, 2017

In an e-mail, Chien told Ars:

Manual attacks are more difficult in the U.S. than in Ukraine based on sheer size. In order to cause an effect, something or someone would need to 'flip the switch', deploy a 'crash' devices, etc., but we don't believe there are any technical hurdles in doing so. Crashoverride, which we saw used in the Ukraine, set the precedence of that. In this case, the actors have the needed access. The day we discover another 'crashoverride v2', it will be too late. That means it was already deployed. We don't expect to see a blackout tomorrow. That will likely require some political event. But it is technically possible.

Asked specifically what was different in the latest attacks, Chien told Ars: "We have seen them perform purposeful activity on operational systems in this case. In the first case, we saw them have what we described as a 'beachhead'. Now, we have seen them penetrate into operational systems and perform activity on those systems. Related, the scale of attempted and successful compromises for the US including the types of organizations affected is much more concerning."

Dragonfly uses a combination of tactics to infect targets. One tactic involved using the publicly available Phishery toolkit to send targets a Microsoft Word document that was programmed to download a template from a predetermined server controlled by the attackers. The server would then query the downloading computer for SMB credentials that many corporate networks use to restrict access to verified users. In many cases, the downloading computers would respond and in the process provide the attackers with the user name and a cryptographic hash to the targeted network. Researchers with Cisco Systems described the so-called template injection attack in July. Once Dragonfly used the password to breach the company's corporate network, the hackers would then traverse to the operational network.

Another Dragonfly infection technique relied on so-called watering hole attacks, in which attackers infected websites known to be frequented by energy company personnel. Dragonfly members would then infect targets when they visited the booby-trapped sites. Yet another tactic was the use of fake Adobe Flash updates that installed backdoors.

Little is known for sure about the people who make up Dragonfly. Text strings embedded into some of their code contains both Russian and French words, an indicator that one or both of those may be false flags intended to deceive investigators. Timestamps found in the malware used in the earlier Dragonfly campaigns suggested the group mostly worked Monday through Friday between what would be the hours of 9 am to 6 pm in Eastern Europe. Timestamps in the malware used in the latest campaign suggested roughly the same hours and region, but the data is far too limited to draw any conclusions. The use of publicly available malware and administrative tools such as PowerShell, PsExec, and Bitsadmin also make attribution difficult.

"What is clear is that Dragonfly is a highly experienced threat actor, capable of compromising numerous organizations, stealing information, and gaining access to key systems," Symantec researchers wrote in Wednesday's report. "What it plans to do with all this intelligence has yet to become clear, but its capabilities do extend to materially disrupting targeted organizations should it choose to do so."

This post was updated to add security experts' reaction to Symantec's findings.