The attacks that recently disrupted website operations at Bank of America and at least five other major US banks used compromised Web servers to flood their targets with above-average amounts of Internet traffic, according to five experts from leading firms that worked to mitigate the attacks.

The distributed denial-of-service (DDoS) attacks—which over the past two weeks also caused disruptions at JP Morgan Chase, Wells Fargo, US Bancorp, Citigroup, and PNC Bank—were waged by hundreds of compromised servers. Some were hijacked to run a relatively new attack tool known as "itsoknoproblembro." When combined, the above-average bandwidth possessed by each server created peak floods exceeding 60 gigabits per second.

More unusually, the attacks also employed a rapidly changing array of methods to maximize the effects of this torrent of data. The uncommon ability of the attackers to simultaneously saturate routers, bank servers, and the applications they run—and to then recalibrate their attack traffic depending on the results achieved—had the effect of temporarily overwhelming the targets.

"This very well could be a kid sitting in his mom's basement in Ohio launching these attacks."

"It used to be DDoS attackers would try one method and they were kind of one-trick ponies," Matthew Prince, CEO and founder of CloudFlare, told Ars. "What these attacks appear to have shown is there are some attackers that have a full suite of DDoS methods, and they're trying all kinds of different things and continually shifting until they find something that works. It's still cavemen using clubs, but they have a whole toolbox full of different clubs they can use depending on what the situation calls for."

The compromised servers were outfitted with itsoknoproblembro (pronounced "it's OK, no problem, bro") and other DDoS tools that allowed the attackers to unleash network packets based on the UDP, TCP, HTTP, and HTTPS protocols. These flooded the banks' routers, servers, and server applications—layers 3, 4, and 7 of the networking stack—with junk traffic. Even when targets successfully repelled attacks against two of the targets, they would still fall over if their defenses didn't adequately protect against the third.

"It's not that we have not seen this style of attacks or even some of these holes before," said Dan Holden, the director of research for the security engineering and response team at Arbor Networks. "Where I give them credit is the blending of the threats and the effort they've done. In other words, it was a focused attack."

Adding to its effectiveness was the fact that banks are mandated to provide Web encryption, protected login systems, and other defenses for most online services. These "logic" applications are naturally prone to bottlenecks—and bottlenecks are particularly vulnerable to DDoS techniques. Regulations that prevent certain types of bank traffic from running over third-party proxy servers often deployed to mitigate attacks may also have reduced the mitigation options available once the disruptions started.

No "root" needed

A key ingredient in the success of the attacks was the use of compromised Web servers. These typically have the capacity to send 100 megabits of data every second, about a 100-fold increase over PCs in homes and small offices, which are more commonly seen in DDoS attacks.

In addition to overwhelming targets with more data than their equipment was designed to handle, the ample supply of bandwidth allowed the attackers to work with fewer attack nodes. That made it possible for attackers to more quickly start, stop, and recalibrate the attacks. The nimbleness that itsoknoproblembro and other tools make possible is often available when DDoS attackers wield tens of thousands, or even hundreds of thousands, of infected home and small-office computers scattered all over the world.

"This one appears to exhibit a fair bit of knowledge about how people would go about mediating this attack," Neal Quinn, the chief operating officer of Prolexic, said, referring to itsoknoproblembro. "Also, it adapts very quickly over time. We've been tracking it for a long time now and it evolves in response to defenses that are erected against it."

Another benefit of itsoknoproblembro is that it runs on compromised Linux and Windows servers even when attackers have gained only limited privileges. Because the DDoS tool doesn't require the almost unfettered power of "root" or "administrator" access to run, attackers can use it on a larger number of machines, since lower-privileged access is usually much easier for hackers to acquire.

Use of Web servers to disrupt online banking operations also underscores the damage that can result when administrators fail to adequately lock down their machines.

"You're talking about a server that has a very lackadaisical security process," said Holden, the Arbor Networks researcher. "Whoever's servers and bandwidth are being used obviously don't realize and understand that they've got unpatched servers and appliances. [The attackers] have compromised them and are taking advantage of that."

State sponsored—or a kid in his basement?

Almost all of the attacks were preceded by online posts in which the writer named the specific target and the day its website operations would be attacked. The posts demonstrate that the writer had foreknowledge of the attacks, but there's little evidence to support claims made in those posts that members of Izz ad-Din al-Qassam Brigades, the military wing of the Hamas organization in the Palestinian Territories, were responsible.

In addition, none of the five experts interviewed for this article had any evidence to support claims the attacks were sponsored or carried out by Iran, as recently claimed by US Senator Joseph Lieberman.

"I don't think there's anything about these attacks that's so large or so sophisticated that it would have to be state sponsored," said Prince, the CloudFlare CEO. "This very well could be a kid sitting in his mom's basement in Ohio launching these attacks. I think it's dangerous to start speculating that this is actually state sponsored."

"Those are big attacks, but they're not so unprecedented that it's worth a press release."

Indeed, the assaults seen to date lack most of the characteristics found in other so-called "hacktivist" attacks, in which attackers motivated by nationalist, political, or ideological leanings strike out at people or groups they view as adversaries. Typical hacktivist DDoS attacks wield bandwidth in the range of 1Gbps to 4Gbps, far less than the 60Gbps torrents seen in these attacks, said Michael Smith, senior security evangelist for Akamai Technologies. Also missing from these attacks is what he called "primary recruitment," in which organizers seek grass-roots supporters and provide those supporters with the tools to carry out the attacks.

"Hacktivists will use many different tools," he explained. "You will see various signatures of tools hitting you. To us, the traffic is homogeneous."

Based on the nimbleness of the attacks, Smith speculated that a disciplined group, possibly tied to an organized crime outfit, may be responsible. Organized crime groups sometimes wage DDoS attacks on banks at the same time they siphon large amounts of money from customers' accounts. The disruption caused by attacks is intended to distract banking officials until the stolen funds are safely in the control of the online thieves.

The cavemen get better clubs

When websites in the late 1990s began buckling under mysterious circumstances, many observers attributed almost super-human attributes to the people behind the disruptions. In hindsight, we know the DDoS attacks that brought these sites down were, as Prince puts it, no more sophisticated than a caveman wielding a club. It's fair to say that the groups responsible for a string of attacks over the past year or so—including the recent attacks on banks—have identified a technical innovation that allows those clubs to pierce current defenses in a way that hadn't been seen before. Such breakthroughs are common in the security world, but more often than not, they're quickly rebuffed by countermeasures assembled by defenders.

More importantly, it's grossly premature to compare these attacks to Stuxnet, the highly sophisticated malware the US and Israel designed to disrupt Iran's nuclear program, or to declare the spate of attacks "Financial Armageddon."

It's also important to remember that DDoS attacks aren't breaches of a bank's internal security. No customer data is ever accessed, and no funds are affected. And while torrents of 60Gbps are impressive, they are by no means historical; CloudFlare's Prince said that he sees attacks of that magnitude about once a month.

"Those are big attacks," he said, "but they're not so unprecedented that it's worth a press release."