In recent years, malware authors have developed increasingly sophisticated rootkits that burrow into the operating system itself, modifying basic filesystem and process management code in a way that ensures they are essentially invisible to anyone using the machine: no files visible, no processes apparent. While some progress has been made in detecting when a rootkit has compromised a system, preemptively blocking an attack has remained challenging, since the malware relies on important system functions. A team of computer scientists have now described a tool, called Hook Safe, that uses virtualization to preempt rootkits by moving and protecting the kernel functions that they target.

Rootkits burrow their way into an operating system's kernel using a process called hooking. The services provided by a kernel—file system and hardware access, memory management, etc.—are accessible through callable functions. The kernel keeps track of where the functions reside in memory using pointers, which contain the address in memory of the function. Hooking involves replacing a legitimate function pointer with one provided by malware. So, for example, the malware might replace (or hook) a file system function with one that behaves perfectly normally except when it comes to the areas of the filesystem where the malware lives; in that case, it returns information that suggests the files aren't there. Any software that uses the kernel for filesystem access will never know the rootkit is present.

Obviously, the simplest way of blocking a rootkit would be to prohibit this process by marking kernel memory as read-only. But there are two problems with this approach. For starters, the ability to perform a kernel hook has many legitimate uses, such as when a new input device hooks into the portions of the kernel that handle mouse or keyboard input. The other problem is that the function pointers are scattered around the kernel's memory footprint, and are sometimes created and destroyed as the kernel creates new objects, like networking sockets. Locking the entire kernel down as read-only would cripple the operating system.

(The authors call this problem the "protection granularity gap." It's possible to lock down a page of memory that contains function pointers but, in doing so, you invariably lock down some dynamic data, which causes problems. There is currently no technology that provides a fine enough granularity to lock down the parts of a memory page that contain the pointers.)

The authors tackle these two problems separately. To identify legitimate kernel hooks, the authors run a clean version of the operating system (in this case, Ubuntu 8.04) under a modified version of the QEMU emulator. Their modified version of QEMU tracks all the kernel hooks that take place in the course of normal operations, and generates a unique signature for each of them. That allows this activity to be recognized and allowed during normal operations.

They solve the granularity gap by creating a shadow copy of every kernel hook they've identified, all collected in contiguous memory pages that can be marked as read-only and protected by a custom version of the Xen hypervisor. Their original location in the kernel gets replaced by a jump statement that shifts execution to what they call a trampoline, which bounces execution to the safe shadow copy, and then returns it to the original execution point. The variable-sized x86 instructions make this a bit more challenging than it might otherwise be, but the authors manage to compensate.

If the hook is used simply to execute the function, everything should take place as normal. If it's used to replace the hook, the protected memory page invokes the Xen hypervisor. Their modified version checks the signature of the action, comparing it against the list generated when the kernel ran under QEMU. If the activity is recognized, it's allowed to go forward. If not, the shadow copy of kernel hooks is kept unmodified.

The authors solved a number of other potential problems in their paper. For example, they block writing to hardware registers, and limit Direct Memory Access transfers to kernel space. Hooks in dynamically allocated kernel objects are tracked using a modified version of the kernel's memory allocator. The fact that the system only becomes active after the kernel is loaded into memory is solved by identifying a few key kernel globals, and walking the memory tree below them.

How does it all work? The authors tested a variety of Linux rootkits, and found that all of them failed on a system protected by Hook Safe, some being unable to infect the target machine, the rest remaining visible to a user after infection. For a variety of typical processes, the overhead of the system was negligible. The worst performance occurred when unzipping a file or using the Apache webserver under heavy load. Both of these require the allocation of lots of memory within the kernel, which invoked some of the authors' code in addition to the normal allocation routines. Still, the worst case was only a six percent performance hit.

The biggest potential problem here, which is recognized by the authors, is that their database of acceptable hooks will end up being incomplete. This is already a problem in the lab, but could be a nightmare in the real world, where software updates and new drivers may appear on a monthly basis. Still, it's easy to envision systems that update the profile of legitimate hooks as part of a software update process, or provides users with the opportunity to approve changes.

A paper describing Hook Safe will be presented at the ACM Conference on Computer and Communications Security.

Listing image by Flickr user lucianvenutian