Background

During the OpenZFS summit last year (2016), Dan Kimmel and I quickly hacked together the zpool checkpoint command in ZFS, which allows reverting an entire pool to a previous state. Since it was just for a hackathon, our design was bare bones and our implementation far from complete. Around a month later, we had a new and almost complete design within Delphix and I was able to start the implementation on my own. I completed the implementation last month, and we’re now running regression tests, so I decided to write this blog post explaining what a storage pool checkpoint is, why we need it within Delphix, and how to use it.

Motivation

The Delphix product is basically a VM running DelphixOS (a derivative of illumos) with our application stack on top of it. During an upgrade, the VM reboots into the new OS bits and then runs some scripts that update the environment (directories, snapshots, open connections, etc.) for the new version of our app stack. Software being software, failures can happen at different points during the upgrade process. When an upgrade script that makes changes to ZFS fails, we have a corresponding rollback script that attempts to bring ZFS and our app stack back to their previous state. This is very tricky as we need to undo every single modification applied to ZFS (including dataset creation and renaming, or enabling new zpool features).

The idea of Storage Pool Checkpoint (aka zpool checkpoint ) deals with exactly that. It can be thought of as a “pool-wide snapshot” (or a variation of extreme rewind that doesn’t corrupt your data). It remembers the entire state of the pool at the point that it was taken and the user can revert back to it later or discard it. Its generic use case is an administrator that is about to perform a set of destructive actions to ZFS as part of a critical procedure. She takes a checkpoint of the pool before performing the actions, then rewinds back to it if one of them fails or puts the pool into an unexpected state. Otherwise, she discards it. With the assumption that no one else is making modifications to ZFS, she basically wraps all these actions into a “high-level transaction”.

In the Delphix product, the scenario is a specific case of the generic use case. We take a checkpoint during upgrade before performing any changes to ZFS and rewind back to it if something unexpected happens.

Usage

All the reference material on how to use zpool checkpoint is part of the zfs(1m) man page. That said, this section demonstrates most of its functionality with a simple example.

First we create a pool and some dummy datasets:

$ zpool create testpool c1t1d0 $ zfs create testpool/testfs0 $ zfs create testpool/testfs1 $ zpool list testpool NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT testpool 7.50G 194K 7.50G - - 0% 0% 1.00x ONLINE -

As you can see, the zpool list command has a new column called CKPOINT that is currently empty. Then we take a checkpoint and perform some destructive actions:

$ zpool checkpoint testpool $ zfs destroy testpool/testfs0 $ zfs rename testpool/testfs1 testpool/testfs2 $ zfs list -r testpool NAME USED AVAIL REFER MOUNTPOINT testpool 109K 7.27G 23K /testpool testpool/testfs2 23K 7.27G 23K /testpool/testfs2 $ zpool list testpool NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT testpool 7.50G 290K 7.50G 88.5K - 0% 0% 1.00x ONLINE - $ zpool status testpool pool: testpool state: ONLINE scan: none requested checkpoint: created Sat Apr 22 08:46:54 2017, consumes 88.5K config: NAME STATE READ WRITE CKSUM testpool ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 errors: No known data errors

After checkpointing the pool, destroying the first dataset and renaming the second one, we see that zpool list and zpool status have some new information for us. The CKPOINT column has been populated now and is the same number as the one displayed by zpool status in the checkpoint row. This amount of space has been freed in the current state of the pool, but we keep it around because it is part of the checkpoint (so we can later rewind back to it if we choose to).

To take a look at the checkpointed state of the pool without actually rewinding to it we can do the following:

$ zpool export testpool $ zpool import -o readonly=on --rewind-to-checkpoint testpool $ zfs list -r testpool NAME USED AVAIL REFER MOUNTPOINT testpool 129K 7.27G 23K /testpool testpool/testfs0 23K 7.27G 23K /testpool/testfs0 testpool/testfs1 23K 7.27G 23K /testpool/testfs1 $ zpool export testpool $ zpool import testpool $ zfs list -r testpool NAME USED AVAIL REFER MOUNTPOINT testpool 115K 7.27G 23K /testpool testpool/testfs2 23K 7.27G 23K /testpool/testfs2

To rewind the whole pool back to the checkpointed state, discarding any change that happened after it was taken:

$ zpool export testpool $ zpool import --rewind-to-checkpoint testpool $ zfs list -r testpool NAME USED AVAIL REFER MOUNTPOINT testpool 129K 7.27G 23K /testpool testpool/testfs0 23K 7.27G 23K /testpool/testfs0 testpool/testfs1 23K 7.27G 23K /testpool/testfs1

To discard the checkpoint when it is not needed anymore:

$ zpool checkpoint --discard testpool $ zpool list testpool NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT testpool 7.50G 289K 7.50G - - 0% 0% 1.00x ONLINE -

Caveats

Taking a checkpoint of the whole state of the pool and rewinding back to it is a heavy-weight solution and imposing some limits to its usage was required to ensure that users cannot introduce any unrecoverable errors to their storage pools. Pools that have a checkpoint disallow any change to their vdev configuration besides the addition of new vdevs. Specifically, vdev attach/detach/remove , mirror splitting, and reguid are not allowed while you have a checkpoint. The hypothetical example is that we wouldn’t be able to recover a pool where a rewind was attempted and the config expected to find data in a vdev that was removed after the checkpoint was taken. Similar reasoning was used for the other aforementioned operations. As for vdev add , although it is supported, the user has to re-add the device in the case of a rewind.

Taking a checkpoint while a device is being removed is also disallowed. Again, the checkpoint references the whole state of the pool including the state of any ongoing operations. Device removal for top-level vdevs is a vdev config change that can span multiple transactions. It would be wrong to rewind back to the checkpoint and attempt to finish the removal of a device that has been removed already.

Another important note for pools that have a checkpoint is the fact that scrubs do not traverse any checkpointed data that has been freed in the current state of the pool. Thus, in the case of errors, these data cannot be detected from the current state of the pool ( zdb(1m) can actually detect them, but cannot repair them).

Finally, due to a technical reason that I will go through in a different post, dataset reservations may be unenforceable while a checkpoint exists and the pool is struggling to find space.

I hope that this post was helpful. Another post should be out soon with more technical details about how this feature is implemented and hopefully answer why the above caveats currently exist. Feel free to email me with any questions or feedback.