Going where few password crackers have gone before, a team of security consultants has deployed a cracking-optimized computer that's completely submerged in mineral oil. Members say the setup offers significant cost savings compared with the same machine that uses air to stay cool.

The rig contains two AMD Radeon 6990 graphics cards, long considered a workhorse for password crackers. While the parallel processing in just one of these $800 cards can make as many as 9 billion password guesses each second (see PC3 in the graph at the bottom of this page), the performance comes at a price. GPUs run extremely hot, particularly when combined with other graphics cards, which drives up the cost of keeping them cool enough to run without burning out. The dedicated fans normally used to keep them cool also generate plenty of noise.

Employees of security consultancy KoreLogic recently deployed the password cracker at Midas Green Tech, an Austin, Texas-based data center that specializes in so-called immersion-cooled server hosting. Unlike the other air-cooled systems KoreLogic uses to test the strength of clients' password policies, the cost of hosting it is less than $60 per month, compared to about $100 for an air-cooled system, said Rick Redman, one of the KoreLogic penetration testers who deployed the new machine.

"I've got this machine. It's overheating," Redman said. "It's pumping out all this power. I don't want to run it in my house because it's too noisy and my wife complains. It's noisy like you wouldn't believe." What's more, it would require about $60 per month worth of electricity to run it at home, he estimated.

Submerging the cards in mineral oil is "quote unquote green," Redman told Ars. "Because I don't have to worry about air flow, I can compact them together and make them so much tighter, so I can save space and use less energy and pay less money. It's cheaper and it's better and it's safer for my computer. It's illogical for me not to do it."

When the same machine was air-cooled, Redman said it was "overheating drastically," even though it ran the GPUs at one of their factory-set speeds as opposed to being "overclocked to run faster." The Radeon 6990, as opposed to many other AMD GPU models, is known to run hot.

KoreLogic recently deployed the immersion-cooled machine and posted video demos to YouTube:

“People just freak out”

To be clear, engineers have been using immersion for decades to cool specialized hardware. The Cray-2 supercomputer developed in the 1980s by Cray Research ran so hot it was submerged in cooling liquid known as fluorinert. Computer game aficionados have long used water to keep their overclocked rigs from melting down. But it's only been in the last few years that immersion cooling has been used in commercial data centers—and even then by only a select few—said Chris Boyd CTO of Midas Green Technologies.

"People just freak out," he said, describing the reactions many still have to the concept of immersion cooling. "We've had electrical engineers who are afraid of mineral oil because they can see the blinking lights. They know there's live AC power in there. Psychologically, it's just a huge step for people to take their box and put it in liquid because we've been told from the time we're small children: 'Don't put anything electrical in the sink.' It's a big culture shift."

Mineral oil doesn't conduct electricity, so it's completely safe to drop a powered-on computer into a tank and even plug in an ethernet cable. A few modifications are required before taking the plunge, though. One involves removing all the fans. Another is replacing the thermal paste that binds a CPU's heatsink to the motherboard with a special foil that absorbs the temperature of the cooled liquid. Redman says immersion-cooled systems work best with a solid state drive, although there are water-tight enclosures for traditional hard drives for those who need them.

Since immersion cooling was introduced about two years ago, Midas has consolidated 30 servers down to just three. Running fewer machines to achieve the same amount of work isn't the only source of cost savings. It turns out it's less expensive to use mineral oil to cool computers. The air-cooled side of the Midas data center relies on a 10hp motor to move air throughout the floor, and that doesn't include compressors, condensers, and other motors needed to make the air conditioning work. By contrast, the immersion side of the house, where about one-third of the data center's 75 servers run, performs all required cooling with just three small motors that together produce less than 1hp.

Redman said KoreLogic has long used souped-up GPU computers to test the strength of the passwords employees of its Fortune 500 and Fortune 50 customers use to secure e-mail and intranet accounts. The ability of a machine that can make billions of guesses per second is augmented by lists made up of hundreds of millions or even billions of real-world passwords and programming rules that extend the reach of the lists. Armed with these setups, crackers frequently decipher as much as 90 percent of leaked password lists, although cracking passwords used in corporate environments is often much harder because employees are required to choose much stronger passcodes.

Graphics cards are key to achieving such results. Unlike a computer's main CPU, GPUs excel in so-called parallel computing, in which many calculations are carried out simultaneously.

"I used to run a system with four 6990s," Jeremi Gosney, a password security expert who has built far bigger cracking behemoths, told Ars. "Since it generated so much heat, I couldn't have it in a computer case; it had to be open-air. But even running two 1200-watt power supplies and having supplemental power connected to the PCI-e bus, I still caught the motherboard on fire after about three months of continuous use. That was when we decided to sell all our 6990s, because they were just too much of a hassle."

Gosney, the CEO and founder of Stricture Consulting Group, said he achieves much better results with the 7900 series of AMD's Radeon GPUs. Given the success of using conventional air to keep his systems cool, he has never been tempted to seek out more exotic techniques.

"We moved on to 7970s, which are a little bit slower than the dual-GPU 6990s for most algorithms, but they are half the cost," he explained. "And you can put four 7970s in a single server chassis and cool them just fine and still have some headroom for overclocking."

Redman said the GPUs in his immersion-cooled system typically run from 58 degrees to 80 degrees Celsius when fully engaged. (Because of a temporary glitch, the temperatures shown in one of the videos are incorrect, he said.) If they were air cooled, they would run in the mid 80s to low 90s, depending on the temperature in the room. While these temperatures may sound high to some gamers, they're actually normal for password cracking. That's because GPUs produce much more heat and consume a lot more power when used for computing as opposed to generating graphics. Brute-forcing fast hashes is particularly hard on GPUs because it hammers the cards' arithmetic logic units.

In addition to building denser systems and running them for less, immersion cooling has the added benefit of being much quieter, particularly when the systems are running GPUs.

"Typically, the GPU boxes have these small, very fast fans in them that shriek and will set your teeth on edge," said Boyd, the Midas CTO. Using water pumps to circulate mineral oil through the systems, by contrast, "is much quieter. If you're in the facility and the cooling pump comes on, it's not even as loud as a cat purring."

Report updated to add detail about solid state drives.