Cryptographically Verifying Container Linux at Runtime

• By Matthew Garrett

Container Linux by CoreOS ships dm-verity, a technology that builds on trusted boot and secure boot to make it impossible for attackers to modify the underlying filesystem containing the OS. This security mechanism is enabled by default, helping ensure that the whole system is in a trustworthy state.

A core part of Container Linux is the automated image-based update strategy. Each Container Linux install has three partitions that are used by the OS:

ROOT, which contains local configuration and data and is mounted at /. If this partition is wiped it will be automatically restored to the initial configuration, making it easy to perform a “factory reset” of Container Linux systems.

USR-A, which contains the operating system itself and is mounted at /usr. This contains the initial version of the operating system after install.

USR-B, which is initially blank. On system update, the new version of the OS is written to USR-B. On post-update reboot, it will be used as /usr instead of USR-A, and the next update will be written to USR-A instead.

Since all local configuration and data is stored in /, and since updates are written out to a separate partition, this means that there’s no need for /usr to be modified at runtime. In fact, the filesystem is explicitly configured such that the kernel will only permit it to be mounted read-only. However, this does nothing to protect against out-of-band modification such as writing directly to the partition. This is where dm-verity comes in.

What is dm-verity

dm-verity is a Linux kernel feature originally developed by Google as part of the Chrome OS project. It interfaces with the kernel’s Device Mapper layer, which allows code to be interposed between the underlying block device (in this case the /usr partition) and the filesystem layer’s view of the partition. When a filesystem is created with dm-verity, each block of the filesystem is hashed and a tree of those hashes stored. Whenever a block is read from the underlying block device, dm-verity ensures that it hashes to the stored value before passing it to the filesystem layer. If the hash doesn’t match, an I/O error is returned instead. As a consequence, any filesystem modifications will generate errors.

For this to be secure, dm-verity also needs to be able to verify that the stored hashes have not themselves been tampered with. This is achieved by storing a root hash and passing that to the kernel when setting up dm-verity at runtime. If any hashes have been modified, they will fail to match the root hash and again an error will be generated.

But what secures the root hash?

We chose to place the root hash inside the kernel image itself, which allows it to be protected by either Trusted Boot or Container Linux’s forthcoming UEFI Secure Boot support. The bootloader extracts the hash after verifying or measuring the kernel and passes it on the kernel command line, allowing it to be used during dm-verity setup. For this to be possible, the root hash has to be stored in the uncompressed region of the kernel. We do this by taking advantage of an interesting historical quirk.

In the early days of the kernel, a kernel image could be written directly to a floppy disk and booted from the firmware. This was achieved by embedding a small bootloader in the kernel’s setup code. As time passed, this became less useful (kernels got too big to fit on floppies, and floppy drives pretty much vanished) and the code was removed. However, in order to alert anybody who was used to this functionality, a stub was added that simply prints:

Use a bootloader

Remove disk and press any key to reboot…

This string is not used in any other part of the boot process. It is also (including newline characters) exactly 64 bytes long. This is, conveniently, the length of an ascii-encoded dm-verity root hash. The hash is generated during OS build and embedded in the kernel before the kernel is signed and its measurements calculated.

Trustworthiness with dm-verity enabled in Container Linux

This ensures that the root hash cannot be tampered with, and allows users to have a high level of certainty in the trustworthiness of their /usr filesystems. We’ve been shipping with dm-verity enabled by default since the 1214.0 release, providing additional security in an entirely transparent manner. This is enabled by default and should be entirely transparent to users, but file an issue if it appears to cause any problems.

If you are interested in learning more about Container Linux in general, join a live webinar about Staying Ahead of Vulnerabilities with Automatic Updates in Container Linux on January 26 at 9:15 a.m. PT, or watch it any time after.