I’m ATM working on ReShifter—a tool for backing up and restoring Kubernetes clusters—and in the context of this work I came across a few things related to etcd that did cost me some cycles to sort out, so I thought I share it here to spare you the pain ;)

In general, the etcd v2 to etcd v3 migration story is well documented, see this blog post here as well as the official docs. Here are a couple of things to be aware of, both from a CLI perspective (i.e. when using etcdctl ) as well as from an API perspective (i.e. moving from the Go client lib v2 to v3):

The v2 data model is a tree, that is, a key identifies either a directory (potentially serving as the root for a sub-tree) or a leaf in the tree, in which case the payload can actually be a value. A key can not at the same time be a leaf note and a directory. In v3, the data model has been flattened, that is, there’s no hierarchy information anymore available amongst entries. So, while you can pretend that, the following is true, in v3 you really are dealing with (flat) key ranges:

/kubernetes.io/namespaces/kube-system --> /kubernetes.io

└──namespaces

└── kube-system

One consequence of the data model change is that code that queries and manipulates etcd2 and etcd3 looks different. In the former case, you can, for example utilize the hierarchy information to recursively traverse the tree; in case of etcd3 you effectively determine the range (start and end key, pretty similar to what you’d do in HBase) and then iterate over the result set; see for example discovery.Visit2() and discovery.Visit3() in the ReShifter code base.

and in the ReShifter code base. There is a difference between the wire-protocol used (HTTP vs. gRPC) and the API version/data model in use. For example, you might have an etcd3 server running, but using it in an etcd2 mode. Be aware of how you’ve configured etcd and in which mode you’re communicating with it.

One thing really caused me some pain: forgetting to set the environment variable ETCDCTL_API=3 . This unremarkable switch causes etcdctl to switch from talking v2 to v3. Run etcdctl before and after setting the env variable and compare the commands you’ve got available, for example get/set in v2 vs. get/put in v3 (see also the screen shot at the top of this post, showing that ls is only available in the v2 API).

. This unremarkable switch causes to switch from talking v2 to v3. Run before and after setting the env variable and compare the commands you’ve got available, for example in v2 vs. in v3 (see also the screen shot at the top of this post, showing that is only available in the v2 API). In an etcd3 server, the v2 and v3 data stores exist in parallel and are independent, see also the terminal session, below.

Let’s have a look now at a simple interaction with etcd3 and how to use the v2 and v3 API. First we launch etcd3, containerized:

Now, let’s put a value into etcd, using the v2 API:

Next, we switch to the v3 API:

$ export ETCDCTL_API=3

And now, we first check if we can read the value we’ve previously set using the v2 API:

Which returns empty, so no way to write to the etcd2 datastore and read it out via v3. Now, let’s put something into etcd using the v3 API and query it right after it to confirm the write: