Since the release of the vSphere CSI driver in vSphere 6.7U3, I have had a number of requests about how we plan to migrate applications between Kubernetes clusters that are using the original in-tree vSphere Cloud Provider (VCP) and Kubernetes clusters that are built with the new vSphere CSI driver. All I can say at this point in time is that we are looking at ways to seamlessly achieve this at some point in the future, and that the Kubernetes community has a migration design in the works to move from in-tree providers to the new CSI driver as well.

However, I had also seen some demonstrations from the Velero team on how to use Velero for application mobility. I wanted to see if Velero could also provide us with an interim solution to move applications with persistent storage between a K8s cluster running on vSphere using the in-tree VCP and a native K8s cluster that uses the vSphere CSI driver.

Note that this method requires downtime to move the application between clusters, so the application will be offline for part of this exercise.

It should also be noted the Cassandra application that I used for demonstration purposes was idle at the time of backup (no active I/O), so that should also be taken into account.

tl;dr Yes – we can use Velero for such a scenario, in the understanding that (a) you will need resources to setup the new CSI cluster and (b) there is no seamless migration, and that the application will need to be shutdown on the VCP cluster and restarted on the CSI cluster. Here are the detailed steps.

External S3 object store

The first step is to setup an external S3 object store that can be reached by both clusters. Velero stores both metadata and (in the case of vSphere backups using restic) data in the S3 object store. In my example, I am using MinIO as I have had the most experience with that product. I have a post on how to set this up on vSphere if you want to learn more. In my lab, my VCP K8s cluster is on VLAN 50, and my CSI K8s cluster is on VLAN 51. Thus, for the CSI cluster to access the MinIO S3 object store, and thus the backup taken from the VCP cluster, I will need to re-IP my MinIO VMs to make the backup visible to the CSI cluster. More detail on that later.

VCP StorageClass

Before going any further, it is probably of interest to see how the VCP driver is currently being used. The reference to the provider/driver is placed in the StorageClass. Here is the StorageClass being used by the Cassandra application in the VCP cluster, which we will shortly be backing up and moving to a new cluster.

$ kubectl get sc NAME PROVISIONER AGE cass-sc kubernetes.io/vsphere-volume 64d $ cat cassandra-sc-vcp.yaml kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: cass-sc provisioner: kubernetes.io/vsphere-volume parameters: diskformat: thin storagePolicyName: raid-1 datastore: vsanDatastore

Deploy Velero on VCP K8s cluster

At this point, the S3 object store is available on IP 192.50.0.20. It is reachable via port 9000. Thus when I deploy Velero, I have to specify this address:port combination in the s3Url and publicUrl as follows:

$ velero install --provider aws --bucket velero \ --secret-file ./credentials-velero \ --use-volume-snapshots=false \ --use-restic --backup-location-config region=minio,s3ForcePathStyle="true",\ s3Url=http://192.50.0.20:9000,publicUrl=http://192.50.0.20:9000

If you are unsure how to deploy Velero on vSphere, there is a post on the Velero blog on how to achieve this . Now you are ready to backup a stateful application. If the install is successful, you should see something similar to this at the end of the output:

Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

Prepping the Stateful Application

For the purposes of this test, I deployed a Cassandra stateful set with 3 replicas on my VCP cluster. I also populated it with some data so that we can verify that it gets successfully restored on the CSI cluster.

$ kubectl exec -it cassandra-0 -n cassandra -- nodetool status Datacenter: DC1-K8Demo ====================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.16.5.2 104.38 KiB 32 66.6% 8fd5fda2-d236-4f8f-85c4-2c57eab06417 Rack1-K8Demo UN 172.16.5.3 100.05 KiB 32 65.9% 6ebf73bb-0541-4381-b232-7f277186e2d3 Rack1-K8Demo UN 172.16.5.4 75.93 KiB 32 67.5% 0f5387f9-149c-416d-b1b6-42b71059c2fa Rack1-K8Demo $ kubectl exec -it cassandra-0 -n cassandra -- cqlsh Connected to K8Demo at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 }; cqlsh> use demodb; cqlsh:demodb> CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint); cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (100, 'Cormac', 'Cork', 999, 1000000); cqlsh:demodb> select * from emp; emp_id | emp_city | emp_name | emp_phone | emp_sal --------+----------+----------+-----------+--------- 100 | Cork | Cormac | 999 | 1000000 (1 rows) cqlsh:demodb>

Backup Cassandra using Velero

We are now ready to backup Cassandra. The first part is to annotate the volumes so that restic knows that it needs to copy the contents of these volumes as part of the backup process.

$ kubectl -n cassandra annotate pod/cassandra-2 \ backup.velero.io/backup-volumes=cassandra-data pod/cassandra-2 annotated $ kubectl -n cassandra annotate pod/cassandra-1 \ backup.velero.io/backup-volumes=cassandra-data pod/cassandra-1 annotated $ kubectl -n cassandra annotate pod/cassandra-0 \ backup.velero.io/backup-volumes=cassandra-data pod/cassandra-0 annotated $ velero backup create cassandra-pks-1010 --include-namespaces cassandra Backup request "cassandra-pks-1010" submitted successfully. Run `velero backup describe cassandra-pks-1010` or `velero backup logs cassandra-pks-1010` for more details.

Commands such as velero backup describe cassandra-pks-1010 and velero backup describe cassandra-pks-1010 –details can be used to monitor the backup. All going well, the backup should complete and at the end of the output from the –details command, you should observe the following.

Restic Backups: Completed: cassandra/cassandra-0: cassandra-data cassandra/cassandra-1: cassandra-data cassandra/cassandra-2: cassandra-data If you logon to the MinIO object store, you should see a new backup called cassandra-pks-1010 . You can also run a velero backup get to check on the full list of backups: $ velero backup get NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR cassandra-pks-1010 Completed 2019-10-10 10:38:32 +0100 IST 29d default <none> nginx-backup Completed 2019-10-01 15:12:20 +0100 IST 21d default app=nginx At this point, you may want to double-check that the application was backed up successfully by attempting to restore it to the same VCP cluster that it was backed up from. I am going to skip such a step here and move straight onto the restore part of the process. Switch contexts to the Kubernetes CSI cluster My current kubectl context is set to my VCP cluster. Let’s switch to the CSI cluster.