Hello and welcome to my new tutorial series on writing a bare metal hypervisor for 64-bit Arm systems!

In this series we will cover the fundamentals of virtualization in the 64-bit Arm architecture and develop an accompanying hypervisor step-by-step, beginning with simple examples hosting trivial virtual machines and working up to hosting fully-fledged operating system kernels such as Linux. Topics discussed will include device tree spoofing & stage 2 address translations, CPU feature spoofing & instruction emulation, device emulation, virtual interrupts, firmware interactions, and more.

The hypervisor code is being actively developed as an open source project on GitHub and will be demoed throughout the series on the Armv8-A Foundation Platform model as shipped with the freely available DS-5 Community Edition. Supplementary videos will also be hosted on my new YouTube channel.

This first post aims to introduce some key concepts and then kick-starts development with an example of how to boot a 64-bit Arm processor into a hypervisor and print “Hello, world!” over a PL011 UART serial console, all in A64 assembly.

What is a hypervisor?

First, take a step back and consider that the role of an operating system kernel is to:

provide layers of abstraction on top of the underlying hardware;

allocate and schedule resources between multiple user-space applications, including execution time, memory, file I/O, and access to peripherals;

In this way the kernel is said to supervise the user-space applications running on top of it. A hypervisor does exactly the same thing, but instead of hosting multiple user-space applications it hosts multiple operating system kernels; it supervises the supervisor, hence the name hypervisor.

A hypervisor creates virtual machines that behave as though they are real, physical machines; in certain use cases the operating system kernel controlling the virtual machine, referred to as a guest, truly believes it has control of a real, physical machine and has no idea it is being virtualized. However this typically comes with a performance penalty so in other use cases the guest is made aware that it is being virtualized and can then cooperate with the hypervisor to reduce overhead; this is known as para-virtualization.

Typical applications of virtualization include sandboxing certain processes & tasks or running a second operating system on a machine that is already running some other operating system, such as to test cross-platform software or to run a legacy application. Virtualization is a widely used tool and is the cornerstone of compute hosting services where one rents a virtual machine running in the cloud (think: Amazon Web Services and Microsoft Azure, to name but a few).

The industry defines two types of hypervisor, imaginatively named Type 1 and Type 2.

Type 1 hypervisors are also known as bare metal hypervisors as they sit directly on the hardware with no underlying host operating system. In contrast, Type 2 hypervisors are also known as hosted hypervisors as they either run on top of, or are themselves a part of, a host operating system running on the hardware: