The long, painful rollout of patches to a security flaw in the Bourne Again Shell (bash) has left thousands of systems still vulnerable, and malware based on the vulnerability continues to spread, according to a number of security experts. But even for organizations that have already applied the patch for what has been dubbed the “Shellshock” vulnerability, the cleanup may not be over—and it could be long and expensive.

Soon after the Shellshock bug was publicly disclosed and its initial patch was distributed, weaknesses in the patch itself and additional security vulnerabilities were uncovered by developers dealing with the issue. And within a day of the disclosure, attacks exploiting the vulnerability were found in the wild. Some of those attacks are still trying to spread—and in some cases, they’re using Google searches to help them find potential targets. Successful attacks may have made changes to the targeted systems that would not have been corrected by the application of the patch.

The problem with Shellshock is similar to problems that emerged after the Heartbleed bug and numerous other vulnerabilities—while organizations struggle to understand the disclosures, how they affect their systems, and how to successfully implement patches, others—including security researchers—race to build proof-of-concept attacks based on them to demonstrate exactly how dire they are. And those proofs of concept often get picked up by cybercriminals and others with bad intent before organizations can effectively patch them—using them to exploit systems in ways that are much longer-lasting than the vulnerability du jour.

It’s this sort of thing that Jason Healy, the director of the Cyber Statecraft Initiative at the Atlantic Council and the author of Wiley’s Cyber Security Policy Guidebook, compared in a presentation at DEF CON 22 this past August to the “gas fight” scene in the movie Zoolander. While ostensibly trying to help secure the Internet, some security researchers may just be throwing gasoline on the problem and adding a spark.

“You can’t just release stuff,” Healy said in an interview with Ars. “When you discover your vulnerability, you need to as a community have more ownership in the consequences that follow. You have to have some maturity in mitigating the results of those consequences. We (in security and cyber research) are still mixed in owning up to those responsibilities in the way we want the government to own up to them.”

Sleeper shell

The “bug” behind Shellshock was actually introduced as a feature 25 years ago by the original lead developer of bash, Brian Fox. On August 5, 1989—long before Linus Torvalds ever considered creating his own kernel—Fox committed code to the bash source that allowed the exporting of functions. Every version of bash since 1.03 has contained the vulnerability, so it has spread far and wide through Unix and Posix-compliant operating systems—making it even more insidious than the OpenSSL Heartbleed vulnerability.

There’s been no indication that the bug was ever exploited over that time “in the wild,” however. It even existed in “hardened” versions of Linux and Unix operating systems used by the government, though other controls on those systems would have prevented it from being exploited effectively. It wasn’t until Shellshock was publicly disclosed that it became a real threat.

The initial execution of the disclosure of Shellshock was done by the book, with most of the affected operating systems ready to push patches to solve it before public disclosure. The problem was that the patches that were pushed out carried new security flaws similar to the old one, and there was some confusion about what types of systems were vulnerable to the bug being exploited. Within hours of the disclosure, however, a number of people had already figured out how to exploit it, and they were doing so in spades.

Who wants to party with the botnet?

Many of them were security researchers who were actively scanning the Internet for vulnerable systems. But there were as many, if not more, malicious exploits in the works attempting to plant all manners of bot scripts, backdoors, and rootkits onto systems that they discovered were vulnerable. Many of them were using IRC-based remote control similar to the bots discovered by Jonathan Hall of Future South Technologies on the servers of WinZip and Yahoo.

Andrew Hay, senior security research lead at OpenDNS, told Ars that within hours of the Shellshock disclosure, the OpenDNS Labs team saw a spike in traffic scanning for the vulnerability in the fraction inbound to OpenDNS customers. For the first two hours, Hay said, it was clear most of the traffic was people trying to determine what was vulnerable. But then, the exploit became “weaponized,” he said, and there was a dramatic rise in the traffic. “The scans were created in an automated fashion to exploit any and all possible (Shellshock) vulnerabilities,” he said.

“One of our researchers is a data scientist,” Hay said, “and we tapped him with finding out which scanning IP addresses were humans versus bots. We looked at all the IP addresses and their other DNS requests.”

OpenDNS spotted a number of known IRC servers being used for botnet control, associated with “shellbot” samples discovered (among other places) on Pastebin. Over the next few days, Hay says, his team saw a sustained amount of DNS requests for those servers—about a thousand per hour—over several days, as the bots were spread by the exploit. Over the first week, two of these domains had more than 5 million DNS requests between them as bots connected for the first time after exploit.

Aside from these more traceable attacks, there are other potential exploits of Shellshock that might be less obvious on security scans, Tatu Ylönen, the inventor of Secure Shell (SSH) and the chief innovation officer of SSH Communications Security, said in a statement emailed to Ars that Shellshock may have been used to give attackers long-lasting remote access to targeted systems running SSH. “Shellshock can be used to inject a Secure Shell key, which can then remain unnoticed for years if there is no management of these keys,” he wrote. “This emphasizes the importance of properly scanning and managing SSH keys. “

Even those who aggressively patched with the fixes released on the first few days could have been at risk of being exploited—even as they struggled to understand which of their systems were vulnerable. Mo Rosen, chief operating officer of the network security company Xceedium, said that his company pushed out a detection tool for Shellshock to customers last week, “driven by a wave of requests from our customers. “We’ve seen very quick response to apply whatever shellshock remediations were available at any given time,” he said. “One customer, within the first two days after the announcement, patched thousands of Unix machines within the first two days after the NIST announcement.”

But because of the gap between an actual complete fix for the vulnerability and its disclosure, Rosen said, just scanning for the vulnerability isn’t enough. In the wake of Shellshock and Heartbleed before it, he said, “I think what we see is this ramped up focus on operating environments where you have to assume you’re not maybe sort of kind of going to get breached—you have to assume you’ve been breached. You [need to] operate with the mindset of a world where you’ve been partially breached all of the time.”

Listing image by B9TRIBECA