The Flame malware that was likely spawned by a nation-state to spy on Iran employed a highly sophisticated cryptography attack that allowed it to pierce defenses Microsoft added to later versions of its Windows operating system, new research shows.

The "chosen prefix collision attack," which exploited known weaknesses in the aging MD5 cryptographic hashing algorithm, was used to remove text strings from counterfeit certificates the attackers used to hijack the Windows Update process. If the critical extension had been allowed to remain in the certificates, they would have caused machines running Vista and later versions of Windows to reject the updates, Microsoft researchers said in a report published Wednesday.

The counterfeit certificates, which were minted by exploiting weakness in Microsoft's Terminal Server product, worked only against versions that predated Vista. But by using the collision attack to remove the "Microsoft Hydra" extension from the certificate's cryptographic hash, they were able to trip up machines running Vista, Server 2008, and Windows 7 as well. In a separate report also published Wednesday, a Kaspersky researcher said the technique gave Flame powerful control over machines running Microsoft's most fortified operating systems.

"What we've found now is better than any zero-day exploit," Alex Gostev, chief security expert at Kaspersky Lab, wrote. "It actually looks more like a 'god mode' cheat code—valid code signed by a keychain originating from Microsoft."

Slaying Hydra

Separate capabilities in Flame were designed to allow the espionage malware to spread from machine to machine inside a victim's network. It worked by setting up a fake server that masqueraded as a legitimate source for Windows updates. For the proxies to work they needed to include the imprimatur of Microsoft's root authority key, and that's where the fraudulent certificates came in. By exploiting weaknesses in the way Terminal Server issued end-user licenses, the Flame attackers were able to create certificates that were authorized by Microsoft's sensitive root to verify their malicious code was legitimate.

But the Flame attackers had one more hurdle to jump through: Credentials ultimately derived from the Terminal Server exploit still contained the Hydra extension, and that flag in turn would cause Vista and later Windows versions to reject the certificate. To remove the extension, they relied on the highly esoteric collision attack, in which two different plaintext sources generate the same cryptographic hash. They used that attack to generate a similar looking certificate that removed the Hydra data and other fields constraining its permitted use. A 2008 exploit that used that technique allowed researchers to create a rogue certificate authority that was trusted by all browsers.

"Without this collision attack, it would have been possible to sign code that would validate on systems pre-dating Windows Vista, but that signed code would fail validation on Windows Vista and above," Jonathan Ness, of Microsoft Security Response Center Engineering, wrote in Wednesday's blog post. "After this attack, the attacker had a certificate that could be used to sign code that chained up to the Microsoft Root Authority and worked on all versions of Windows."

As previously reported, Microsoft on Sunday issued an emergency update to all Windows users that invalidated the entire certificate chain used by the Terminal Server licensing mechanism. In the Wednesday post, Ness said Microsoft has replaced the mechanism's chain with a new hierarchy that's no longer linked to the company's Root Authority. Instead, it has a stand-along root that's not trusted by the rest of Windows. The certificates use SHA-1, an algorithm that cryptographers consider much stronger than MD5. Microsoft has also curtailed the practice of issuing code-signing certificates under the licensing regimen.

Ness's post never explained one of the biggest mysteries arising from the Flame aftermath, which is why Microsoft engineers designed the old system with such poor key management. The Microsoft Root Authority is the cryptographic equivalent of a master key that can unlock virtually any door in the company's sprawling body of software. Tying that authority to Terminal Server's licensing mechanism is tantamount to using a hotel's universal key to control access to the janitor closet.