In the past, passwords were cracked by randomly guessing at the correct string of characters in what's known as a "brute force" attack. In these assaults, the encrypted form of the password - the hash - is extracted from the target file or computer. A randomly generated password is encrypted and its encrypted form is compared to the extracted hash. If it doesn't match, the process is repeated until a match is found - it's a long and tedious process. With rainbow tables, the encrypted form of most possible passwords are pre-computed and stored alongside the actual, clear-text password. Users can simply look up virtually any hash in the massive index and match it to the corresponding password in seconds.

The tables can break password protection in many common file formats, including versions of Adobe's PDF format (the current version is immune to the attack), the default encryption on protected Microsoft Office documents (40 bit) and even Windows password files. "It's a lot of (storage) space but the nice thing is it only needs to be done once," says Pieter Zatko, a division scientist at BBN Technologies, a government contractor that conducts research for the US Department of Defence and other government agencies. Mr Zatko is best known for writing the L0phtcrack password cracking tool in the '90s. It was used to crack Windows passwords with ease, something he hoped would change the way organisations managed their passwords. Instead, L0phtcrack was commercialised and became the industry-standard password auditor, much to Mr Zatko's dismay. "That was my problem with L0phtcrack. People were using it to audit their passwords," he says. "It was supposed to be a statement of 'understand your risks'."

It seems people haven't learnt from his work - passwords are still easy to crack. Neohapsis, for example, uses rainbow tables for forensic investigations, cracking passwords when a client's disgruntled ex-employees refuse to hand them over. The tables can't be used to crack strongly encrypted passwords, but many computer users are lazy, using the same passwords over and over. Thus, obtaining a trivial, easily cracked password could be the foothold a consultant - or a criminal attacker - needs to unravel a user's full set of passwords.

"The scope of the password or pass-phrase problem is only increasing," says Neohapsis' Greg Shipley. "We found a password on a document and that password was used on other documents ... Unless a user is incredibly sophisticated, you crack one of them and there's a good chance you're going to get the rest of them." By attacking the low-hanging fruit of easily cracked document or file passwords, it's possible to guess at more sensitive passwords, such as those used for email or remote network access. "Anything that's 40-bit encrypted you can hit with a rainbow table," Mr Stankevitz says. Although 40-bit is far from the strongest encryption available, it's still in common use because of now-redundant US Government arms trafficking regulations, says Mr Zatko. "For quite some time we had these stupid regulations. Cryptography was considered a munition, so we weren't allowed to export anything that was greater than 40-bit encryption," he says.

It's not just the good guys using rainbow tables, either. "There are services online that you can upload your file to and they'll decrypt it for you for $50, no questions asked," says Mr Stankevitz. The ease with which miscreants can crack file or document passwords means using the same password for your VPN access and password protected files is a very bad idea.

Indeed, Mr Shipley says that a fairly comprehensive rainbow table set is now within reach of hobbyists and criminals alike, and certainly organised crime syndicates. "You can get a terabyte or two of storage in the four figure range and you can get computer power in the four figure range," he says. "You can still do a lot at a hobbyist level, and if you're dealing with organised crime rings they'll have the budgets." In the meantime, the guys at Neohapsis has some advice for Microsoft Office users: change the encryption setting to 128 bit and use a 12-character password.