Piercing a key defense found in cloud environments such as Amazon's EC2 service, scientists have devised a virtual machine that can extract private cryptographic keys stored on a separate virtual machine when it resides on the same piece of hardware.

The technique, unveiled in a research paper published by computer scientists from the University of North Carolina, the University of Wisconsin, and RSA Laboratories, took several hours to recover the private key for a 4096-bit ElGamal-generated public key using the libgcrypt v.1.5.0 cryptographic library. The attack relied on "side-channel analysis," in which attackers crack a private key by studying the electromagnetic emanations, data caches, or other manifestations of the targeted cryptographic system.

One of the chief selling points of virtual machines is their ability to run a variety of tasks on a single computer rather than relying on a separate machine to run each one. Adding to the allure, engineers have long praised the ability of virtual machines to isolate separate tasks, so one can't eavesdrop or tamper with the other. Relying on fine-grained access control mechanisms that allow each task to run in its own secure environment, virtual machines have long been considered a safer alternative for cloud services that cater to the rigorous security requirements of multiple customers.

"In this paper, we present the development and application of a cross-VM side-channel attack in exactly such an environment," the scientists wrote. "Like many attacks before, ours is an access-driven attack in which the attacker VM alternates execution with the victim VM and leverages processor caches to observe behavior of the victim."

The attack extracted an ElGamal decryption key that was stored on a VM running the open-source GNU Privacy Guard. The code that leaked the tell-tale details to the malicious VM is the latest version of the widely used libgcrypt, although earlier releases are also vulnerable. The scientists focused specifically on the Xen hypervisor, which is used by services such as EC2. The attack worked only when both attacker and target VMs were running on the same physical hardware. That requirement could make it harder for an attacker to target a specific individual or organization using a public cloud service. Even so, it seems feasible that attackers could use the technique to probe a given machine and possibly mine cryptographic keys stored on it.

The technique, as explained by Johns Hopkins University professor and cryptographer Matthew Green, works by causing the attack VM to allocate continuous memory pages and then execute instructions that load the cache of the virtual CPU with cache-line-sized blocks it controls. Green continued:

The attacker then gives up execution and hopes that the target VM will run next on the same core—and moreover, that the target is in the process of running the square-and-multiply operation. If it is, the target will cause a few cache-line-sized blocks of the attacker's instructions to be evicted from the cache. Which blocks are evicted is highly dependent on the operations that the attacker conducts.

The technique allows attackers to acquire fragments of the cryptographic "square-and-multiply" operation carried out by the target VM. The process can be difficult, since some of the fragments can contain errors that have the effect of throwing off an attacker trying to guess the contents of a secret key. To get around this limitation, the attack compares thousands of fragments to identify those with errors. The scientists then stitched together enough reliable fragments to deduce the decryption key.

The researchers say it's the first demonstration of a successful side-channel attack on a virtualized, multicore server. Their paper lists a few countermeasures administrators can take to close the key leakage. One is to avoid co-residency and instead use a separate, "air-gapped" computer for high-security tasks. Two additional countermeasures include the use of side-channel resistant algorithms and a defense known as core scheduling to prevent attack VMs from being able to tamper with the cache processes of the other virtual machine. Future releases of Xen already include plans to modify the way so-called processor "interrupts" are handled.

While the scope of the attack remains limited, the research is important because it opens the door to more practical attacks in the future.

"This threat has long been discussed, and security people generally agree that it's a concern," Green wrote. "But actually implementing such an attack has proven surprisingly difficult."