I’m playing with ZFS on Linux using Debian jessie (and eventually stretch). I decided I really wanted to use ZFS for everything, including /boot/grub and swap space. In addition, I wanted to boot using UEFI. This is how I did it.

The official ZFS Root on Debian HOWTO is much more cautious and wants you to create an ext4 partition for /boot/grub and a separate partition for swap. Being the idiot^Wdaredevil I am, I decided to flout that advice and go all-in on ZFS. The rest of this article details the steps involved in making this happen. (You really want to read the ZFS Root on Debian guide first, and then come back here to see where I deviated. Also, standard disclaimers apply: I’m doing this on a laptop I don’t care about and won’t use for any production purpose. If following this guide creates a monster that eats your disk and calls your mother names, I take no responsibility. It worked for me, but might not work for you. Phew.)

Get Your Install Environment Ready

This part is going to smell a lot like the official guide. I’ll make a note of where I deviate from it, though.

Download the rEFInd Boot Manager. You’ll use it to reboot into your new Debian install the first time. Download and boot a Live CD, such as debian-live-8.3.0-amd64-standard.iso. The official guide specifically requires using a Live CD and not just an installer CD, so I used the “standard” non-graphical Live CD. Use whatever Live CD you are comfortable with, as long as it’s Debian 8. (Of course, I’m doing all this with an “iso-hybrid” image that I’ve dd 'ed to a USB stick.) Once booted, download and install the zfsonlinux package. This simply adds the ZoL package repository to your APT sources. $ sudo -i # wget -q http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_6_all.deb # dpkg -i zfsonlinux_6_all.deb Selecting previously unselected package zfsonlinux. (Reading database ... 44337 files and directories currently installed.) Preparing to unpack zfsonlinux_6_all.deb ... Unpacking zfsonlinux (6) ... Setting up zfsonlinux (6) ... Install the ZFS packages in the Live CD environment. This will compile the SPL and ZFS dynamic kernel modules, which will take some time. # apt-get update Hit http://archive.zfsonlinux.org jessie InRelease Hit http://archive.zfsonlinux.org jessie/main amd64 Packages Hit http://security.debian.org jessie/updates InRelease Ign http://archive.zfsonlinux.org jessie/main Translation-en_US Ign http://archive.zfsonlinux.org jessie/main Translation-en Hit http://security.debian.org jessie/updates/main Sources Hit http://security.debian.org jessie/updates/main amd64 Packages Ign http://http.debian.net jessie InRelease Hit http://security.debian.org jessie/updates/main Translation-en Hit http://http.debian.net jessie Release.gpg Hit http://http.debian.net jessie Release Hit http://http.debian.net jessie/main Sources Hit http://http.debian.net jessie/main amd64 Packages Hit http://http.debian.net jessie/main Translation-en Reading package lists... Done # apt-get install linux-image-amd64 debian-zfs [...] Loading new spl-0.6.5 DKMS files... First Installation: checking all kernels... Building only for 3.16.0-4-amd64 Building initial module for 3.16.0-4-amd64 Done. [...] Loading new zfs-0.6.5.2 DKMS files.. First Installation: checking all kernels... Building only for 3.16.0-4-amd64 Building initial module for 3.16.0-4-amd64 Done. [...] Load the kernel module and verify that it loaded properly: # modprobe zfs # dmesg | grep ZFS [ 7446.660850] ZFS: Loaded module v0.6.5.2-2, ZFS pool version 5000, ZFS filesystem version 5

Partition Your Disks

I am performing these steps on a laptop with a single drive; as such my setup will not uncover any issues that may arise from using multiple drives in a mirror or RAIDZ configuration. (Just FYI.)

I want everything on ZFS, including /boot/grub and my swap partition. So I will create one large zpool using the entire disk: # zpool create -o ashift=12 -o altroot=/mnt -m none rpool /dev/disk/by-id/<disk_name> # zfs set atime=off rpool # zfs set relatime=on rpool # zfs set compression=lz4 rpool zpool create will label the disk using GPT and actually create two partitions; partition 1 is the ZFS pool, and partition 9 is for the EFI System Partition (ESP). We’ll use this later. I also go ahead and disable atime , enable relatime , and enable lz4 compression on the root dataset so that these are inherited all the way down. I can always override them in a child dataset if necessary. “Wait, so handing zpool(1m) a whole disk still creates two partitions?”, you ask? Yes. Yes it does. So then why do I care so much about not creating a third or fourth partition for /boot/grub or swap? Principles, my friend. And blatant disregard for rules. Moving on… Use zfs(1m) to create a zvol for swap space: # zfs create -o compression=off -V 4G rpool/swap # mkswap -L swap /dev/zvol/rpool/swap Setting up swapspace version 1, size = 4194300 KiB LABEL=swap, UUID=354f7cba-fa33-45bb-97c7-7c6aa6109b22 I’ll make sure to add an entry into /etc/fstab later, after that file gets created. Create the ROOT filesystem. (This is not the same as / ; this will be the parent dataset for all future boot environments.) # zfs create -o mountpoint=none rpool/ROOT Create the initial “boot environment” and mount rpool at /rpool : # zfs create -o mountpoint=/ rpool/ROOT/debian-1 # zfs set mountpoint=/rpool rpool Setting the mountpoint for the rpool filesystem deviates from the ZFS on Debian guide, but seems in-line with what OpenIndiana does by default. I do this now instead of when I created the pool, because otherwise the rpool/ROOT/debian-1 filesystem cannot mount properly. Set the bootfs property on the pool to the boot environment you just created: # zpool set bootfs=rpool/ROOT/debian-1 rpool Create any other filesystems you want. If the filesystems should be managed as a unit (i.e., all part of the same boot environment), then make sure to create them under rpool/ROOT/debian-1 . (While there’s not a beadm(1m) -type command for Linux yet, I still want to set up the system up as if there were. That way if/when beadm gets ported, the system will be ready to handle it.) # zfs create -o mountpoint=/home rpool/home # zfs create -o mountpoint=/usr rpool/ROOT/debian-1/usr # zfs create -o mountpoint=/var rpool/ROOT/debian-1/var # zfs create -o mountpoint=/var/tmp -o setuid=off rpool/ROOT/debian-1/var/tmp # zfs create -o mountpoint=/tmp -o setuid=off rpool/tmp The ZFS on Debian guide mentions setting exec=off on the two tmp filesystems they create. When I followed those instructions, I got errors during DKMS compilation, I believe, so I left it enabled. Finally, export the zpool. # zpool export rpool

Once all the above steps are done, you should now have the following layout (you might want to check this before exporting the pool):

# zpool get all rpool NAME PROPERTY VALUE SOURCE rpool size 238G - rpool capacity 0% - rpool altroot - default rpool health ONLINE - rpool guid 14661786247243457179 default rpool version - default rpool bootfs rpool/ROOT/debian-1 local rpool delegation on default rpool autoreplace off default rpool cachefile - default rpool failmode wait default rpool listsnapshots off default rpool autoexpand off default rpool dedupditto 0 default rpool dedupratio 1.00x - rpool free 237G - rpool allocated 1.12G - rpool readonly off - rpool ashift 12 local rpool comment - default rpool expandsize - - rpool freeing 0 default rpool fragmentation 0% - rpool leaked 0 default rpool feature@async_destroy enabled local rpool feature@empty_bpobj active local rpool feature@lz4_compress active local rpool feature@spacemap_histogram active local rpool feature@enabled_txg active local rpool feature@hole_birth active local rpool feature@extensible_dataset enabled local rpool feature@embedded_data active local rpool feature@bookmarks enabled local rpool feature@filesystem_limits enabled local rpool feature@large_blocks enabled local # zfs list -t all -o name,type,mountpoint,compress,exec,setuid,atime,relatime NAME TYPE MOUNTPOINT COMPRESS EXEC SETUID ATIME RELATIME rpool filesystem /rpool lz4 on on off on rpool/ROOT filesystem none lz4 on on off on rpool/ROOT/debian-1 filesystem / lz4 on on off on rpool/ROOT/debian-1/usr filesystem /usr lz4 on on off on rpool/ROOT/debian-1/var filesystem /var lz4 on on off on rpool/ROOT/debian-1/var/tmp filesystem /var/tmp lz4 off off off on rpool/home filesystem /home lz4 on on off on rpool/swap volume - off - - - - rpool/tmp filesystem /tmp lz4 off off off on

Install Debian

Reimport the pool and create the cache file: # zpool import -d /dev/disk/by-id -R /mnt rpool # mkdir -p /mnt/etc/zfs # zpool set cachefile=/mnt/etc/zfs/zpool.cache rpool In case you’re wondering about the options passed to zpool import (and don’t have a copy of the zpool(1m) man page), the -d option tells zpool to search the /dev/disk/by-id directory for importable pools, and the -R option tells it to use /mnt as the altroot and set the cachefile property to none . The latter gives us the chance to create the directory for the cache file first, then set the option appropriately. Since I didn’t create a separate partition for /boot/grub , I can skip some things in the official guide and move directly to using debootstrap(8) to install the base Debian system. # apt-get install debootstrap # debootstrap --arch=amd64 jessie /mnt http://httpredir.debian.org/debian/ [...] I: Base system installed successfully.

Configure Your New Debian System

Create /etc/{hosts,hostname} . Substitute whatever you’re going to name your machine for “debzfs” in the command below. # echo debzfs > /mnt/etc/hostname # sed -i -Ee "s#(127.+)#\1 debzfs#" /mnt/etc/hosts # cat /mnt/etc/hosts 127.0.0.1 localhost debzfs ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters Create /mnt/etc/fstab . It should be empty (except for a comment, maybe), but a line needs to be added for the swap device. Here I use printf(1) for terseness, but you can use your favorite editor to do this as well. # printf "/dev/zvol/rpool/swap\tnone\t\tswap\tdefaults\t0 0

" >> /mnt/etc/fstab # cat /mnt/etc/fstab /dev/zvol/rpool/swap none swap defaults 0 0 If you created a dataset for /tmp not under the boot environment (like I did), you might notice at some point that systemd won’t mount /tmp before things want to use it. If that is the case, you can attempt to try the workaround documented here, or you can do what I did and just set the mountpoint on rpool/tmp to legacy and add a line to fstab , like this: # zfs set mountpoint=legacy rpool/tmp # printf "rpool/tmp\t\t/tmp\t\tzfs\tdefaults\t0 0

" >> /mnt/etc/fstab # cat /mnt/etc/fstab /dev/zvol/rpool/swap none swap defaults 0 0 rpool/tmp /tmp zfs defaults 0 0 I’d much rather figure out how to tell systemd to mount /tmp earlier, but until then this method certainly works. Do some minor network configuration. The ZFS on Debian page just puts all interface configuration into /mnt/etc/network/interfaces , but here I’ve chosen to create a separate file per-interface in /mnt/etc/network/interfaces.d/ . It’s your choice. # ls /mnt/etc/network/interfaces.d/ eth0 lo # cat /mnt/etc/network/interfaces.d/eth0 auto eth0 iface eth0 inet dhcp # cat /mnt/etc/network/interfaces.d/lo auto lo iface lo inet loopback Bind-mount some useful filesystems into the new system. # for f in dev dev/pts proc sys; do mount -v --bind {,/mnt}/$f; done mount: /dev bound on /mnt/dev. mount: /dev/pts bound on /mnt/dev/pts. mount: /dev/proc bound on /mnt/proc. mount: /dev/sys bound on /mnt/sys. Go ahead and copy the zfsonlinux package you downloaded earlier into /mnt/root so you don’t have to download it again, then chroot(1) into your new system. # cp zfsonlinux_6_all.deb /mnt/root/ # chroot /mnt /bin/bash --login Now that you’re in the new environment, you have to setup ZoL all over again…. (It seems that there is a documentation error in the official guide when running the locale-gen command. Use what I’ve got below instead.) # apt-get install locales [...] # sed -i -Ee 's/# (en_US.UTF+)/\1/' /etc/locale.gen # locale-gen Generating locales (this might take a while)... en_US.UTF-8... done Generation complete. # apt-get install lsb-release [...] # dpkg -i /root/zfsonlinux_6_all.deb Selecting previous unselected package zfsonlinux. (Reading database ... 11210 files and directories currently installed.) Preparing to unpack zfsonlinux_6_all.deb ... Unpacking zfsonlinux (6) ... Setting up zfsonlinux (6) ... # apt-get update [...]

Okay, hold on. I’m going to stop here and deviate once again from the ZFS on Debian guide.

APT-Pin ZoL Packages

In the realm of “that thing that happened that one time”, the first time I tried this setup I got into a situation after updating to Debian’s testing channel (stretch) where the spl package from Debian updated before the spl and zfs packages from ZoL, which broke ZFS for a while until I could uninstall the Debian version and reinstall the ZoL version. Since then I’ve created an preferences file to pin ZoL packages at a higher priority than the Debian ones. (I’m sure this will never come back to bite me, ever.) If you want to do this, create the file /etc/apt/preferences.d/50-zol-packages with the following content. If you’d rather not mess with the Natural Order of Things, then feel free to skip this part.

# cat <<EOF > /etc/apt/preferences.d/50-zol-packages > Package: * > Pin: origin archive.zfsonlinux.org > Pin-Priority: 900 > EOF # apt-cache policy spl spl: Installed: 0.6.5-1 Candidate: 0.6.5-1 Version table: *** 0.6.5-1 0 900 http://archive.zfsonlinux.org/debian/ jessie/main amd64 Packages 100 /var/lib/dpkg/status # apt-cache policy grub2-common grub2-common: Installed: 2.02-beta2.9-ZOL11-7aa9f6 Candidate: 2.02-beta2.9-ZOL11-7aa9f6 Version table: *** 2.02-beta2.9-ZOL11-7aa9f6 0 900 http://archive.zfsonlinux.org/debian/ jessie/main amd64 Packages 100 /var/lib/dpkg/status 2.02~beta2-22+deb8u1 0 500 http://httpredir.debian.org/debian/ jessie/main amd64 Packages

(Debian jessie doesn’t have an spl package, but stretch (testing) does, so I’m showing the output for grub2-common as well to show you the difference.)

Install ZoL (Again) and GRUB2 EFI Support

And now we continue on. Installing the debian-zfs metapackage will install the spl , spl-dkms , zfs-dkms , and zfsutils packages. Note that the DKMS modules will get compiled (again) at this time.

Once more I’ve deviated from the ZFS on Debian instructions. Instead of installing grub-pc , I’m installing the grub-efi package because I want to boot via UEFI. Exciting times.

# apt-get install linux-image-amd64 debian-zfs [...] # apt-get install grub2-common grub-efi zfs-initramfs [...] # apt-get dist-upgrade [...]

Digression: Get Ready for UEFI Boot

Keep in mind that the Debian live-installer you booted from didn’t boot in (U)EFI mode, so configuring GRUB2 will only halfway work. What we need to do is reboot into UEFI mode so that grub-install can access and set EFI variables properly. However, before we do that we can finish out the installation process and prepare partition 9 to be the ESP. We need to install the dosfstools package to be able to format part9 as VFAT. Note that my disk is /dev/sda ; yours might be different. You can also use the /dev/disk/by-id/identifier-part9 path instead if you want to stay consistent with the rest of this process.

# apt-get install dosfstools # mkdir /boot/efi # mkfs.vfat /dev/sda9 mkfs.fat 3.0.27 (2014-11-12) # mount /dev/sda9 /boot/efi # grub-probe -d /dev/sda9 fat # update-grub Generating grub configuration file ... Found linux image: /boot/vmlinuz-3.16.0-4-amd64 Found initrd image: /boot/initrd.img-3.16.0-4-amd64 done # grub-install -d /usr/lib/grub/x86_64-efi /dev/sda # Note: actually unsuccessful! Installing for x86_64-efi platform. efibootmgr: EFI variables are not supported on this system. efibootmgr: EFI variables are not supported on this system. Installation finished. No error reported # find /boot/efi -type f /boot/efi/EFI/debian/grubx64.efi

We’re going to want to mount the ESP to /boot/efi , so add a line to /etc/fstab for it. (If you want, you can mount it read-only instead, or not automatically mount it, etc.)

# printf "/dev/sda9\t\t/boot/efi\tvfat\tdefaults\t0 1

" >> /etc/fstab # cat /etc/fstab /dev/zvol/rpool/swap none swap defaults 0 0 rpool/tmp /tmp zfs defaults 0 0 /dev/sda9 /boot/efi vfat defaults 0 1

At this point, you will not be able to boot your install without using rEFInd. We’ll fix that in a minute. Let’s go ahead and finish out the instructions in the official guide.

# passwd root Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully # umount /boot/efi # exit logout # for f in dev/pts dev proc sys boot/efi; do umount /mnt/$f; done # zfs umount -a # zpool export rpool

If umount(8) complains about not being able to unmount /sys , find out what other random things got mounted there while you weren’t looking ( mount|grep /mnt/sys ) and unmount those first, then try to unmount /sys again.

Reboot into rEFInd

Download the rEFInd Boot Manager USB image (if you haven’t already) and dd(1) it to a USB stick. (If you have a second thumb drive, use that so that you still have the Debian Live environment on the first stick in case you need it.) Once you’ve done that, reboot your machine. Now would be a good time to make sure your “BIOS” is set to boot into UEFI mode. After verifying that, insert the rEFInd stick and tell your system to boot to it.

Even though grub-install failed to write to the EFI variables, it still put all of the boot files where they needed to go, so rEFInd should be able to identify your Debian installation. Select it from the boot manager screen (if it’s not already selected) and boot.

Log in as root (because everything worked swimmingly and it actually booted, right? Right? Whew!) and use grub-install to fix the UEFI variables it couldn’t set last time.

# grub-install /dev/sda Installing for x86_64-efi platform. Installation finished. No error reported.

Before doing anything else, let’s reboot without rEFInd and make sure that all the UEFI stuff is set properly and that your machine can boot under its own power. If anything goes wrong, go back over this article and make sure that you performed all the steps (and that none of them failed, horribly or otherwise). You should also consult the ZFS Root on Debian documentation as well.

If you rebooted and everything worked, you should be able to log back in as root and start installing whatever other things you want! Before you get too far down the road, though, it’d probably be a good idea to snapshot your boot environment just in case.

# zfs snapshot -r rpool/ROOT/debian-1@2016020401-pristine # zfs list -rt all -o name,type,used,refer,mountpoint rpool/ROOT NAME TYPE USED REFER MOUNTPOINT rpool/ROOT filesystem 1.12G 96K none rpool/ROOT/debian-1 filesystem 1.12G 554M / rpool/ROOT/debian-1@2016020401-pristine snapshot 0 554M - rpool/ROOT/debian-1/usr filesystem 337M 337M /usr rpool/ROOT/debian-1/usr@2016020401-pristine snapshot 0 337M - rpool/ROOT/debian-1/var filesystem 252M 252M /var rpool/ROOT/debian-1/var@2016020401-pristine snapshot 0 252M - rpool/ROOT/debian-1/var/tmp filesystem 96K 96K /var/tmp rpool/ROOT/debian-1/var/tmp@2016020401-pristine snapshot 0 96K -

Conclusion

And this is where I leave you, with Debian installed on a ZFS root and booting from UEFI. Where you go from here is up to you. Install a desktop environment or two, all the utilities you want, create yourself a regular user, and enjoy the benefits of using ZFS.