That’s VME as in Virtual-8086 Mode Enhancements, introduced in the Intel Pentium Processor, and initially documented in the infamous NDA-only Appendix H.

Almost immediately since the Ryzen CPUs became available in March 2017, there have been various complaints about problems with Windows XP in a VM and with running 16-bit applications in DOS boxes in Windows VMs. Multiple versions of Windows are affected. Some (but not all) other operating systems are affected as well, for example OS/2 Warp running in a VM on Ryzen when attempting to open a DOS window:

After analyzing the problem, it’s now clear what’s happening. As incredible as it is, Ryzen has buggy VME implementation; specifically, the INT instruction is known to misbehave in V86 mode with VME enabled when the given vector is redirected (i.e. it should use standard real-mode IVT and execute in V86 mode without faulting). The INT instruction simply doesn’t go where it’s supposed to go which leads to more or less immediate crashes or hangs.

How did AMD miss it? Because only 32-bit OSes are affected, and only when running 16-bit real-mode code. Except with Windows XP and Server 2003 it’s much worse and these systems may not even boot.

To be clear, the problem is not at all specific to virtualization. It has been confirmed on a Ryzen 5 1500X running FreeDOS—which comes with the JemmEx memory manager, which enables VME by default. Until VME was disabled, any attempt to boot with JemmEx failed with invalid opcode exceptions. After disabling VME, FreeDOS worked normally.

That is not surprising because when the problematic INT instruction is executed inside a VM using AMD-V, it is almost always executed without any intervention from the hypervisor, which means the hypervisor has no opportunity to mess anything up.

Now, back to the XP trouble. Windows NT supports VME at least since NT 4.0 and enables it automatically. That is the case for NT 4.0, XP, Windows 7, etc. For the most part, it would only matter when running a 16-bit DOS or Windows application (such as EDIT.COM which comes with Windows).

Windows XP and Server 2003 (that is NT 5.1 and 5.2) is significantly more affected because it was the first Windows OS that shipped with a generic display driver using VBE (VESA BIOS Extensions), and the only Windows family which executed the BIOS code inside NTVDM (with VME on, if available). Starting with Vista, presumably due to increased focus on 64-bit OSes where V86 mode is entirely unavailable, the video BIOS is executed indirectly, likely using pure software emulation.

The upshot is that the problem is visible in Windows versions at least from NT 4.0 and up, but XP and Server 2003 may entirely fail to boot, either hanging or crashing just before bringing up the desktop. Other operating systems which use VME are affected as well (OS/2, DOS with certain memory managers).

The workaround is simple—if possible, mask out the VME CPUID bit (bit 1 in register EDX of leaf 1), which is something hypervisors typically allow. Windows does not require VME and without VME, XP can be booted normally on Ryzen CPUs, at least in a VM.