Code dive A super-bug in the Xen hypervisor may allow privileged code running in guests to escape to the underlying host.

This means, on vulnerable systems, malicious administrators within virtual machines can potentially break out of their confines and start interfering with the host server and other guests. This could be really bad news for shared environments.

All versions of open-source Xen are affected (CVE-2016-6258, XSA-182) although it is only potentially exploitable on x86 hardware running paravirtualized (PV) guests. The bug was discovered by Jérémie Boutoille of Quarkslab, and publicly patched on Tuesday for Xen versions 4.3 to 4.7 and the latest bleeding-edge code.

Xen is used all over the internet, particularly by cloud providers to run customers' virtual machines. If your cloud host has or had emergency downtime this month, this will be why – patching is required. For what it's worth, Amazon says its AWS is not affected. Xen usually shares details of its upcoming advisories under embargo to those who need to know, so patches can be applied before the information is made public and everyone finds out how to exploit the holes.

According to the Xen security team, "the PV pagetable code has fast-paths for making updates to pre-existing pagetable entries, to skip expensive re-validation in safe cases (e.g. clearing only Access/Dirty bits). The bits considered safe were too broad, and not actually safe."

In other words, when guest kernels update the data structures that describe how memory is organized within the virtual machine, Xen should only flip certain flag bits. The name of the game with virtualization is to control and limit what guests can do. In this case, Xen doesn't go far enough to protect bits in the host's page tables that could be changed to open up a guest's access to other parts of the system, in theory.

Developers working on the security-conscious open-source Qubes project, which uses Xen to lock down and wall off applications from each other, are unsure whether or not XSA-182 is fully exploitable. Qubes OS users should, by the way, install updated Xen packages as soon as possible.

"In order to be on the safe side, we decided to continue treating it as critical bug nevertheless," the Qubes team noted in an advisory on Tuesday.

"After all, the bug does violate some fundamental assumptions about immutability of certain memory mapping structures. The mere fact we were unable to come up with an agreeable exploitation sketch within the last 24 hours should not be treated as a mitigation factor."

The programming cockup is extremely similar to October's XSA-148, which also allowed guests to escape to the host.

The problem is that Xen contains code like this...

#define l3_disallow_mask(d) (!is_pv_32bit_domain(d) ? \ base_disallow_mask : 0xFFFFF198U)

...which apart from looking ugly is a code smell: magic numbers invariably mask assumptions and bugs in the design of software and should be avoided. Elsewhere you have slightly more tidier code like this:

if ( !l1e_has_changed(ol1e, nl1e, PAGE_CACHE_ATTRS | _PAGE_RW | _PAGE_PRESENT) ) { ... } if ( !l2e_has_changed(ol2e, nl2e, unlikely(opt_allow_superpage) ? _PAGE_PSE | _PAGE_RW | _PAGE_PRESENT : _PAGE_PRESENT) ) { ... }

There are checks like these for each of the page table levels. These bit masks – (PAGE_CACHE_ATTRS | _PAGE_RW | _PAGE_PRESENT) and (_PAGE_PSE | _PAGE_RW | _PAGE_PRESENT) – are easier to grok by humans (well, humans who like systems programming) but are incomplete. The patches, in part, replace these lines with:

#define FASTPATH_FLAG_WHITELIST \ (_PAGE_NX_BIT | _PAGE_AVAIL_HIGH | _PAGE_AVAIL | _PAGE_GLOBAL | \ _PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_USER) if ( !l1e_has_changed(ol1e, nl1e, ~FASTPATH_FLAG_WHITELIST) ) { ... } if ( !l2e_has_changed(ol2e, nl2e, ~FASTPATH_FLAG_WHITELIST) ) { ... }

Thus ensuring all the checks, not just these two examples, use a common and consistent set of flags that are known to be safe to manipulate. The patch note even says: "Rework the logic as a whitelist, to make it clearer to follow."

We also note that Xen's mm.c source file is 190KB and 6,505 lines in size.

"This bug, being the second critical bug in the Xen PV virtualization code publicly discussed in a relatively short period of time, cannot simply be shrugged off, patched, and forgotten," the Qubes team continued, turning its ire on Xen's engineers.

"It begs for answers to critical questions, such as: Has Xen been written by competent developers? How many more bugs of this caliber are we going to witness in the future? What can or should we do to protect against such gaping holes?"

The answer to that last question, for the Qubes project anyway, is hardware memory virtualization aka second-level address translation aka nested paging, which the upcoming Qubes 4.0 hopes to support. This means Qubes OS shifting to using Xen PVH rather than Xen PV as it does at the moment.

The vast majority, if not all, Intel, AMD and recent ARM Cortex cores provide nested paging, which Qubes OS can use to further separate guests from the underlying host. Nested paging offers two sets of page tables: one that maps guest virtual memory to guest physical memory, and another that maps guest physical memory to host memory. Guests can manipulate the first set to their hearts' content, while the second set is kept separate by the hypervisor and hardware, thus preventing virtual machines from directly screwing with a host CPU's precious page tables.

Nested paging has been around for a while, offers extra security and performance, and is supported by Xen, Hyper-V, KVM, VMware, VirtualBox, Parallels, bhyve, OpenBSD and no doubt other hypervisors.

The Qubes team noted that it has abstracted its code away from being Xen-specific, allowing it to drop in a replacement hypervisor, should one appear.

Lars Kurth, Xen Project champion, told us via email:

Every computer system has bugs and therefore we have an industry-accepted security process in place that notifies significant downstream users in private if a bug is found. We provide a reasonable interval for downstream to respond and prepare updated software packages – that our security team prepares – to our pre-disclosure list; and then make public disclosures.

"We take security very seriously and have developed security process best practices that are aimed for cloud environments that maximize fairness and transparency," he added. ®

PS: There's a couple of other denial-of-service bugs in Xen this week: XSA-183 and XSA-184.