Many of us use software firewalls, virus scanners, and other security software on our PCs. We expect this software to make our computers safer, but some new research suggests that it contains a whole host of exploitable vulnerabilities.

The Matousec researchers found that common software tools, including Norton Internet Security 2010, McAfee Total Protection 2010, and Trend Micro Internet Security Pro all had flaws that allowed attackers to bypass the protections that these programs offer. The malicious software can do this without even having to run as an Administrator.

The common feature of the vulnerable software is that it patches the Windows kernel to enable it to intercept certain operations like opening files or killing processes, a process called hooking. Windows lists all these functions in a table, the System Service Descriptor Table (SSDT), with each function having a number specifying its position in the table. To call a kernel function from nonkernel—user-mode—software, Windows essentially tells the processor to switch into kernel mode and call the function with the desired number. By overwriting entries in the table, the security software can intercept function calls.

In addition to choosing the function to call, the user-mode program must also pass in any necessary data to the function—for example, the name of the file to open. The kernel has very strict rules on which memory it can access; incorrect memory accesses that merely cause a user-mode program to crash will cause the entire kernel to fail, resulting in a system-halting blue screen of death. This means that these hooked functions have to carefully validate the parameters that are passed to them to ensure that they do not refer to any memory that is inaccessible.

After validating the parameters, the security software then typically performs some check such as ensuring that a call to NtTerminateProcess, the API that terminates processes, does not attempt to terminate one of the security software's own programs.

Finally, the hooks call the real functions, so that Windows can actually perform the requested operation. When doing this, the hooks pass in the original parameters unmodified, to ensure that the real kernel function operates correctly.

This is where the vulnerability lies. Programs can change the meaning of the parameters after they pass to hooked functions. If the change is made at just the right time—after the validation, but before they get sent to the original kernel function—the program can give the hook function data that it accepts, but then replace it with data that would be rejected. So, for example, it would make a call to NtTerminateProcess with a reference to a harmless process, and then quickly replace the reference with one to the security software itself. The hook would see the harmless process and permit the operation to continue, but the real kernel function would see the reference to the security software, and duly terminate it.

This requires careful timing on the part of the attacker. The replacement has to be made just at the right moment. Too soon, and the hook will attempt to validate the malicious parameters and reject the call. Too late, and the harmless data will already have been passed on to the real function. This might seem improbable, but it turns out that an attack specially written to target anti-virus and firewall software by using these hooks can successfully switch around its parameters after just a handful of attempts.

The researchers found exploitable versions of this vulnerability in every program they tested, including products from McAfee, Trend Micro, and Kaspersky. In fact, the researchers said that the only reason that they found exploits in only 34 products was that they only had time to test 34 products (Microsoft, for its part, believes that its security software is not affected, but is still investigating the issue). Many others may be vulnerable too. They also developed a toolkit dubbed KHOBE ("kernel hook bypassing engine") to allow the rapid detection and exploitation of such flaws.

Matousec initially believed the technique they were using to exploit the security software was newly discovered. After publication, however, they became aware that the basic technique was documented as a way of attacking Unix way back in 1996. A 2003 posting to security mailing list Bugtraq described using the same technique against Windows.

A matter of timing

However, until relatively recently, successful exploitation was almost impossible. To have time to switch the parameter data around, the attacking program needs to make sure that one of its threads gets a chance to run during the window of vulnerability. On a single-core machine, where only one thread can run at any one time, the probability of a thread switch occurring at just the right time (to allow the replacement to be made) is so low that the software is all but unexploitable. But multicore systems, capable of running multiple threads simultaneously, are a lot more plentiful today than they were in 2003—let alone 1996.

With multiple cores running concurrently, the timing issue is greatly reduced. Though software cannot guarantee that it will be able to replace the data at just the right time, it is free to try again over and over, and after a few attempts it can generally pull off a successful attack. The growth of multicore processors has turned an attack that was mostly theoretical into one that's practical and reliable.

This isn't the first vulnerability found in security software that modifies the SSDT. In 2007 Matousec published a list of flawed programs that had errors in their hooks that allowed attackers to crash machines or even escalate their privileges, enabling malicious software to run with all the privileges that the kernel has.

Though this new, old flaw does not appear to carry this kind of risk—it can't be used to achieve escalated privileges, and attackers must already be able to run software on the victim's system. But it does mean that victims can have their security software disabled at the whims of attackers.

The researchers didn't describe any solutions for the problems in its public paper, though they do have them available to paying customers. The difficulty inherent in any solution is that to ensure that the real function operates properly, the hooks shouldn't modify any of the parameters, but instead must pass the original ones. This is what permits the attacks to be made, but seems hard to avoid.

In particular, what might seem the obvious naive approach—have the hook function copy all the user data into memory that it controls (so it cannot be altered by the malicious software—does not work well. This is because Windows applies different security checks depending on whether a call is made from user-mode, using user-mode memory addresses, or kernel-mode. In addition to performing the validation mentioned previously, user-mode calls are also subject to access control list (ACL) checks. ACLs are used to secure things like files and registry keys, and are an essential part of Windows' security.

Only user-mode calls need to have ACLs checked. Kernel-mode calls do not, because the kernel is privileged and is allowed access to any file on the system. Hooks generally don't perform this kind of verification because getting it right is complicated and error-prone. Moreover, there's no need: as long as the hooks pass the original parameters to the real function, the real function will do all that work for them. The complexity of these checks means that the naive approach is itself risky, and hence a poor solution to the problem.

A 64-bit solution?

One thing that might be a solution is switching to 64-bit Windows. 64-bit Windows largely prohibits this kind of kernel modification with a system called PatchGuard. With PatchGuard, any attempt to update the SSDT will result in the machine blue-screening shortly thereafter. On the face of it, PatchGuard should be a perfect cure, since it forces security software vendors to use different techniques such as using Windows' built-in filtering mechanisms.

Unfortunately for end-users, a number of security companies—the same security companies that routinely write insecure, exploitable kernel hooks—complained about PatchGuard, claiming that it prevented them from writing effective security software for 64-bit operating systems. Microsoft eventually relented, offering an API to allow certain patching operations to be performed by third parties.

The API that Microsoft offered in response to the security software vendors' demands is not public, so whether it permits SSDT patching is uncertain. Given the consistently flawed attempts to hook the SSDT, and the difficulty in making those hooks robust, a good case can be made that such operations should remain prohibited. Furthermore, not all security software uses SSDT hooks, so clearly this risky technique can be avoided altogether.

In spite of its limitations, this research reveals an effective technique for crippling a wide range of security software. It is unfortunate that Matousec could not test more software so that we could better gauge just how widespread the problem is, and see if there's anything on the market that avoids it. As things stand, though, it appears that even unprivileged malware can readily disable the firewall and anti-virus software that many people depend on, and that can't be a good thing.