One of the papers presented at the Black Hat USA 2008 security conference was an analysis a number of the protection mechanisms built into Windows Vista and Windows Server 2008 that are designed to make it harder to convert software bugs into security flaws. How to Impress Girls with Browser Memory Protection Bypasses, authored by security researchers Mark Dowd at IBM and Alexander Sotirov at VMware, presented a number of attacks against Vista's various security features in isolation, and then attacks that could disable multiple protections all together. Put together, the result is that Vista's mitigation mechanisms are circumvented, making buggy software exploitable.

The security features being bypassed are all intended to minimize the impact of buffer overflows. Buffer overflows are a particular kind of programming error that occur when a program attempts to store too much data in the buffer allocated for the data. This causes anything following the buffer to be overwritten. Buffer overflows are exploitable when it's possible to insert arbitrary executable code into a process and then make that code run. If an attacker can do this then the attacker has gained the ability to do whatever he likes to the victim's computer.

Mitigating against buffer overflows

This kind of flaw is quite a common one, especially in the programming languages C and C++. Many high-profile software flaws have been of this type, from the Morris worm of the 1980s to the Code Red worm of 2001, and more recently the animated cursor vulnerability. Although there are languages that make such flaws impossible—Java and .NET are both immune to such flaws—the unfortunate reality is that a large proportion of the software that we run (including our operating systems, web browsers, and browser plugins) don't use these safe languages, and so are susceptible to this ancient problem.

This is why Microsoft included a number of protection schemes in Vista to try to reduce the exploitability of buffer overflows. Although the operating system cannot prohibit such flaws (except by mandating the use of Java and .NET), it can make it less likely that an overflow can lead to arbitrary code execution. One of these protections was introduced in Windows XP Service Pack 2; Microsoft calls it Data Execution Protection, DEP. With DEP enabled each block of memory in a process must be explicitly marked "executable" before the processor can run any instructions stored in that block. This means that even if an attacker can write arbitrary code into a process, the processor isn't able to run that code. This effectively prevents any easy exploitation of buffer overflows.

Unfortunately, security researchers are a clever lot; they discovered ways by which DEP could be defeated, for example by passing control not to their own executable code, but instead to one of the system DLLs loaded into the process and getting that to do their dirty work. Vista therefore introduced several mechanisms to try to reduce the impact of these DEP bypasses. One of these is Address Space Layout Randomization, which randomly organizes the location of the system DLLs so that an attacker no longer knows where they are. Vista also inserts extra checks into the operating system code to detect that certain kinds of overflow have occurred and crash the program (although crashing might seem a bad thing to do, it's safer than continuing to run after a buffer overflow).

It is these extra mechanisms that the paper at Black Hat attacks. Dowd and Sotirov describe several different techniques for bypassing Windows' protections that can be used to reinstate the exploitability of buffer overflows. One of the key mechanisms used is the fact that the protections are not always applied. Internet Explorer 7 and Firefox 2 both opt out of DEP, and many third-party libraries such as the Flash plugin opt out of ASLR (and other protection mechanisms). Plugins can also do things that can deliberately defeat the OS's countermeasures; Java, for example, marks all of its memory as executable, meaning that a Java applet can place into memory executable code that's immune to DEP protection. The final trick is to use scripting or plugins to file large amounts of memory with the malicious executable code, so that even when ASLR is in effect, an attacker can still be sure that the malicious code is where he needs it to be. Together, these techniques allow all of the protections found in Vista to be defeated.

Chicken Little runs amok

This is certainly unfortunate. The great thing about these protection mechanisms is that they provided a degree of safety even when applications contained bugs. That will no longer be the case, at least for web browsers (programs that do not support third-party plugins (or apply more stringent checks to those plugins) might continue to benefit from the protections). Unfortunate, yes, but not—as was reported in the immediate aftermath of the presentation—evidence that Vista's security is useless, nor does this work constitute a major security issue. And it's not game over, either. Sensationalism sells, and there's no news like bad news, but sometimes—particularly when covering security issues—it would be nice to see accuracy and level-headedness instead. Alarmism helps no one. Responsible vulnerability disclosure is a big concern in the security industry; it would be good to see it coupled with responsible reporting.

The work done by Dowd and Sotirov focuses on making buffer overflows that were previously not exploitable on Vista exploitable. These are buffer overflows that would be exploitable on Windows XP anyway; after all, there's no need to defeat ASLR if an OS does not have ASLR at all. Furthermore, these attacks are specifically on the buffer overflow protections; they do not circumvent the IE Protected Mode sandbox, nor Vista's (in)famous UAC restrictions. DEP, ASLR, and the other mitigation features in Vista are unlikely to ever be unbreakable, especially in an application like a web browser that can run both scripts and plugins of an attacker's choosing. Rather, their purpose is to make exploitation more difficult. Microsoft has a solution for those wanting to make it impossible—use .NET. These protections are there for when that's not an option, to reduce—but not eliminate—the vulnerability caused by such programming errors. Even with DEP and ASLR, the coding errors that result in buffer overflows still ought to be fixed; it is only through fixing the errors that the flaws can truly be eliminated.

Even with the attacks described in the paper, Vista has many worthwhile security improvements compared to XP. Internet Explorer on Vista runs in a highly restricted environment, so that even when it is running malicious code it cannot harm the system. Stories suggesting that Vista's security is now irredeemably broken are far off the mark; the truth is merely that some of its automatic security protection is less effective than it was before.

What Microsoft will do in response remains to be seen. Some of the specific featurs of the attacks can be resolved by Microsoft itself—preventing IE plugins from opting out of the protection schemes, by improving the way that .NET interacts with the protection, and by making Windows default to enabling all the protection schemes—and others can be minimized by third parties—by writing plugins that enable with all the security mechanisms, by being more careful with executable memory, and so on. Longer term, a switch to 64-bit programs might allow considerably more randomization to be applied; while making large allocations is enough to fill up a 32-bit program's memory (which allows attackers to defeat randomization) the same is not true of 64-bit processes—they're simply too big.