Consistently backup your virtual machines using libvirt and zfs – part 1

How to backup virtual machines is a pretty interesting topic and a lot could be said. COW file systems like zfs or btrfs actually do most of the job for you, thanks to their snapshotting capabilities. Unfortunately that’s not enough to get consistent backups, because taking a snapshot of a running VM is very similar to unplugging the power cord. In most cases this isn’t as bad as it sounds, but it is extremely bad if you’re running databases or so. That means you will get corrupt data, which is something we want to avoid at all costs. But how to avoid that? Shutting down the machines before taking the snapshots could be a solution, but that’s only viable if you do daily backups at most. What if we want hourly snapshots? That’s simply unfeasible. The next thing we could do is to pause the VM, take a snapshot of the disk, dump the ram and the EFI vars and then resume the guest. That would be way better, but it still involves some kind of downtime. Is it possible to get it any better? If you use qcow2 you could use its internal snapshotting features to do live snapshots of the state of the machine, but that unfortunately doesn’t work anymore if you use UEFI and it’s also not so well maintained. Also you probably want to use ZVOLs, so no way.

The best alternative out there are libvirt external snapshots. They allow you to freeze the VM image (be it a raw file, qcow2 or zvol), take a dump of the ram and then keep writing all subsequent writes to an external qcow2 file. Actually we don’t really need the external qcow2 file at all, because we can use zfs to track the diff instead. It means that as soon as we created the libvirt snapshot we can immediately take a zfs snapshot and then merge back the external file into the original image.

I use sanoid to take the zfs snapshots and I wanted to keep using it. Unfortunately it didn’t support pre/post scripts, but luckily there were some patches floating around. They didn’t expose all the things I needed in order to get it working, so I made my own fork where I forward ported the patch to latest git master, plus adding additional features to get all the data I needed: https://github.com/darkbasic/sanoid

If you’re using Arch Linux here is a PKGBUILD which tracks my branch, with the addition of systemd timers which the AUR package didn’t have: sanoid-git.tar

Let’s see how it’s implemented:

zfs list

rpool/VM 36.0G 357G 24K none

rpool/VM/fedora28 9.04G 41.0G 8.21G /var/lib/libvirt/images/fedora28

rpool/VM/win2k16 26.9G 73.1G 17.3G /var/lib/libvirt/images/win2k16

rpool/VM_snapshots 34K 357G 34K /var/lib/libvirt/snapshots



As you can see I have a dataset called VM which contains an additional dataset for each VM. There I also store the nvram with the EFI VARS, because it’s important to backup them as well. Additionally I have another dataset called VM_snapshots which I use to store the external qcow2 diff. Its only purpose is to avoid that it gets snapshotted along with the rest of the machine: we don’t need it and it will cease to exist a few seconds later.

Here is my sanoid config:

[rpool/VM]

use_template = production,scripts

recursive = yes

# if you want sanoid to manage the child datasets but leave this one alone, set process_children_only.

process_children_only = yes

[template_production]

hourly = 36

daily = 30

monthly = 3

yearly = 0

autosnap = yes

autoprune = yes

[template_scripts]

### run script before snapshot

### dataset name will be supplied as an environment variable $SANOID_TARGET

pre_snapshot_script = /opt/scripts/prescript.sh

### run script after snapshot

### dataset name will be supplied as an environment variable $SANOID_TARGET

post_snapshot_script = /opt/scripts/postscript.sh

### don't take an inconsistent snapshot

#no_inconsistent_snapshot = yes

### run post_snapshot_script when pre_snapshot_script is failing

#force_post_snapshot_script = yes

This is the content of my prescript:

#!/bin/bash

DOMAIN=${SANOID_TARGET##*/}

SNAPSHOT_NAME=${SANOID_SNAPNAME}

RAM_BACKUP=/mem

# Backup xml

cp /etc/libvirt/qemu/${DOMAIN}.xml /var/lib/libvirt/images/${DOMAIN}/

# Find out if running or not

STATE=`virsh dominfo $DOMAIN | grep "State" | cut -d " " -f 11`

if [ "$STATE" = "running" ]; then

# Take a libvirt snapshot

virsh snapshot-create-as ${DOMAIN} ${SNAPSHOT_NAME} \

--diskspec vda,snapshot=external,file=/var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.disk.qcow2 \

--memspec file=/var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.mem.qcow2,snapshot=external \

--atomic

fi

exit 0

Again, you will need my fork of sanoid in order to get pre-post scripts supports and in particular the additional environment variables. Hopefully soon it won’t be necessary anymore.

What’s going on? First we check if the machine is running, because if it isn’t a regular zfs snapshot will be enough. If it’s running, on the other hand, we do an external libvirt snapshot and we dump the memory.

Now all subsequent writes will go through the external qcow2 and sanoid will take the zfs snapshot.

This is the content of my postscript:

#!/bin/bash

DOMAIN=${SANOID_TARGET##*/}

SNAPSHOT_NAME=${SANOID_SNAPNAME}

RAM_BACKUP=/mem

# Find out if running or not

STATE=`virsh dominfo $DOMAIN | grep "State" | cut -d " " -f 11`

if [ "$STATE" = "running" ]; then

# Commits content from top images into base and adjust the base image as the current active image (--pivot)

virsh blockcommit ${DOMAIN} vda --active --wait --pivot

# Delete snapshot

rm /var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.disk.qcow2

# Once the 'blockpull' operation above is complete, we can clean-up the tracking of snapshots by libvirt to reflect the new reality

virsh snapshot-delete ${DOMAIN} ${SNAPSHOT_NAME} --metadata

# Move the ram to a bigger and cheaper drive.

mkdir ${RAM_BACKUP}/${DOMAIN} 2> /dev/null

mv /var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.mem.qcow2 ${RAM_BACKUP}/${DOMAIN}/

fi

exit 0

As soon as the snapshot is taken we want to merge the external qcow2 file back to the original image using blockcommit. We don’t need it because zfs will take care of the diff. Now it’s time to backup our precious ram dump. We don’t want to waste our Optane 3D XPoint memory with it, so it will get stored on a slower and cheaper drive.

What’s next? We still need more sanoid hooks, in particular pre/post pruning scripts because we want to delete our ram dumps when the old snapshots get deleted. I will probably implement it sooner or later, but since I don’t know Perl patches are welcome.

We also want to send/receive our snapshots to an off site machine (zfs snapshots are not backups), but that’s for part 2!

If you want to further look into the topic I suggest you to read the following:

https://www.spinics.net/lists/virt-tools/msg11470.html

https://wiki.libvirt.org/page/I_created_an_external_snapshot,_but_libvirt_will_not_let_me_delete_or_revert_to_it

https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

https://blog.programster.org/kvm-external-snapshots

https://www.redhat.com/archives/libvirt-users/2013-October/msg00018.html

https://kashyapc.fedorapeople.org/virt/lc-2012/lceu-2012-virt-snapshots-kashyap-chamarthy.pdf

https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-illustration.txt

https://kashyapc.fedorapeople.org/virt/lc-2012/live-backup-with-external-disk-snapshots-and-blockpull.txt

https://iclykofte.com/kvm-live-online-backups-external-and-internal/

https://wiki.libvirt.org/page/Live-merge-an-entire-disk-image-chain-including-current-active-disk