Researchers have devised a technique that bypasses a key security protection built into just about every operating system. If left unfixed, this could make malware attacks much more potent.

ASLR, short for "address space layout randomization," is a defense against a class of widely used attacks that surreptitiously install malware by exploiting vulnerabilities in an operating system or application. By randomizing the locations in computer memory where software loads specific chunks of code, ASLR often limits the damage of such exploits to a simple computer crash, rather than a catastrophic system compromise. Now, academic researchers have identified a flaw in Intel chips that allows them to effectively bypass this protection. The result are exploits that are much more effective than they would otherwise be.

Nael Abu-Ghazaleh, a computer scientist at the University of California at Riverside and one the researchers who developed the bypass, told Ars:

ASLR is an important defense deployed by all commercial Operating Systems. It is often the only line of defense that prevents an attacker from exploiting any of a wide range of attacks (those that rely on knowing the memory layout of the victim). A weakness in the hardware that allows ASLR to be bypassed can open the door to many attacks that are stopped by ASLR. It also highlights the need for CPU designers to be aware of security as part of the design of new processors.

An Intel spokesman said he was investigating the research paper.

Abu-Ghazaleh and two colleagues from the State University of New York at Binghamton demonstrated the technique on a computer running a recent version of Linux on top of a Haswell processor from Intel. By exploiting a flaw in the part of the CPU known as the branch predictor, a small application developed by the researchers was able to identify the memory locations where specific chunks of code spawned by other software would be loaded. In computer security parlance, the branch predictor contains a "side channel" that discloses the memory locations.

When branches collide

A table in the predictor called the "branch target buffer" stores certain locations known as branch addresses. Modern CPUs rely on the branch predictor to speed up operations by anticipating the addresses where soon-to-be-executed instructions are located. They speculate whether a branch is taken or not and, if taken, what address it goes to. The buffers store addresses from previous branches to facilitate the prediction. The new technique exploits collisions in the branch target buffer table to figure out the addresses where specific code chunks are located.

Nothing's stopping malicious attackers from bundling a similar bypass app with attack code that exploits a critical OS or application vulnerability. The exploit could then use the disclosed memory location to ensure malicious payloads are successfully executed by a targeted computer, instead of being flushed without ever being run, as is normally the case when ASLR is active. The researchers believe that ASLR implemented by both Microsoft Windows and Apple's OS X is similarly vulnerable. They have yet to perform research on other chip architectures to see if they also contain side channels that defeat ASLR.

The branch target buffer side channel isn't the only method for bypassing ASLR. Exploits often circumvent the protection by exploiting a second so-called memory-disclosure vulnerability present in the targeted OS or application. The advantage with the new technique is that attackers need not identify a second weakness. The new attack also works across virtualization boundaries, allowing the attack to be carried out in cloud environments, for example. The new technique is also faster and more nimble than an older form of ASLR bypass known as JIT spraying.

On Tuesday, the researchers presented the bypass at the IEEE/ACM International Symposium on Microarchitecture in Taipei, Taiwan. Their accompanying paper, titled "Jump Over ASLR: Attacking the Branch Predictor to Bypass ASLR," proposes several hardware and software approaches for mitigating attacks.