Those of you who work in the cloud native space will probably be aware of VMware’s acquisition of Heptio back in December 2018. Heptio bring much expertise and a number of products to the table, one of which I was very eager to try it. This is the Heptio Velero product, previously known as Heptio Ark. Heptio Velero provides a means to back up and restore cloud native applications. Interestingly enough, they appear to be able to capture all of the deployment details, so they are able to backup the pods (compute), persistent volumes (storage) and services (networking), as well as any other related objects, e.g. statefulset. They can back up or restore all objects in your Kubernetes cluster, or you can filter objects by types, namespaces, and/or labels. More details about how Velero works can be found in heptio.github.io.

Velero can be used with both cloud and on-prem deployments of K8s. These cloud providers (Amazon, Google, etc) have snapshot providers available for the creating snapshots of the application’s Persistent Volumes (PVs). On-prem is a little different. There is a provider available for Portworx. However, for everybody else, there is the open source “restic” utility which can be used for snapshotting PVs on other platforms, including vSphere.

Velero also ships with the ability to create/deploy a container based Minio object store for storing backups. Note that out of the box, this is very small, and won’t let you back up very much. I had to add a large PV to the Minio deployment to make it useful (I’ll show you how I did that shortly). The other issue I encountered with Minio is that the Velero backup/restore client (typically residing in a VM or on your dekstop) will not be able to communicate directly with the Minio object store. This means you can’t look at the logs stored in the object store from the velero client. Thus, a workaround is to change the Minio service from using a ClusterIP to use a NodePort instead. This will provide external access to the Minio object store. From that point, you can update one of the Minio configuration YAML files to tell the client about the external access to the Minio store. From that point, you can display the logs etc. I’ll show you how to do that as well in the post.

My K8s cluster was deployed via PKS. This also has a nuance (not a major one) which we will see shortly.

1. Creating a larger Minio object store

After doing some very small backups, I quickly ran out of space in the Minio object store.The symptoms were that my backup failed very quickly, and when I queried the podVolumeBackups object, I saw the following events:

cormac@pks-cli:~$ kubectl get podVolumeBackups -n velero NAME AGE couchbase-backup-c6bbl 4m couchbase-backup-lsss6 3m couchbase-backup-p9j4b 2m cormac@pks-cli:~$ kubectl describe podVolumeBackups couchbase-backup-c6bbl -n velero Name: couchbase-backup-c6bbl Namespace: velero . . Status: Message: error running restic backup, stderr=Save(<lock/f476380dcd>) returned error, retrying after 679.652419ms: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. Save(<lock/f476380dcd>) returned error, retrying after 508.836067ms: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. Save(<lock/f476380dcd>) returned error, retrying after 1.640319969s: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. Save(<lock/f476380dcd>) returned error, retrying after 1.337024499s: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. Save(<lock/f476380dcd>) returned error, retrying after 1.620713255s: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. Save(<lock/f476380dcd>) returned error, retrying after 4.662012875s: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. Save(<lock/f476380dcd>) returned error, retrying after 7.092309877s: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. Save(<lock/f476380dcd>) returned error, retrying after 6.33450427s: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. Save(<lock/f476380dcd>) returned error, retrying after 13.103711682s: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. Save(<lock/f476380dcd>) returned error, retrying after 27.477605106s: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. Fatal: unable to create lock in backend: client.PutObject: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed. : exit status 1 Path: Phase: Failed Snapshot ID: Events: <none> cormac@pks-cli:~$

Therefore to do anything meaningful, I modified the Minio deployment so that it had a 10GB PV for storing backups. To do it, I created a new StorageClass, a new PersistentVolumeClaim, and the modified the configuration in config/minio/00-minio-deployment.yaml to dynamically provision a PV rather than simply use a directory in the container. Since my deployment is on vSAN and using the vSphere Cloud Provider (VCP), I can also specify a storage policy for the PV. Note the namespace entry for the PVC. It has to be in Velero, along with everything else in the configuration for Velero. The full deployment instructions found at heptio.github.io.

1.1. New StorageClass

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: minio-sc provisioner: kubernetes.io/vsphere-volume parameters: diskformat: thin storagePolicyName: gold datastore: vsanDatastore

1.2. New PersistentVolumeClaim

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: minio-pv-claim-1 namespace: velero annotations: volume.beta.kubernetes.io/storage-class: minio-sc spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi

1.3. config/minio/ 00-minio-deployment.yaml changes

apiVersion: apps/v1beta1 kind: Deployment metadata: namespace: velero name: minio labels: component: minio spec: strategy: type: Recreate template: metadata: labels: component: minio spec: volumes: - name: storage persistentVolumeClaim: claimName: minio-pv-claim-1 - name: config emptyDir: {}

The change here is to change the volume “storage” from using an emptyDir to instead use a Persistent Volume. It is dynamically provisioned via the PersistentVolumeClaim entry. The StorageClass and PVC YAML can be added to the Minio configuration YAML to make things easier for any additional redeploys. Once the new configuration is in place, and once you redeploy the configuration, Minio will now have a 10GB PV available to place backups and allow you to do something meaningful.

2. Accessing Logs

Any time I tried to display backup logs, it would error and complain about not being able to find Minio. It would complain with a “dial tcp: lookup minio.velero.svc on 127.0.0.1:53: no such host” or something to that effect. This is because Minio is using ClusterIP and is not available externally. To resolve this, you can edit the service, and change the ClusterIP to a NodePort.

cormac@pks-cli:~/Velero$ kubectl edit svc minio -n velero

Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"component":"minio"},\ "name":"minio","namespace":"velero"},"spec":{"ports":[{"port":9000,"protocol":"TCP","targetPort":9000}],\ "selector":{"component":"minio"},"type":"ClusterIP"}} creationTimestamp: 2019-03-06T11:16:55Z labels: component: minio name: minio namespace: velero resourceVersion: "1799322" selfLink: /api/v1/namespaces/velero/services/minio uid: 59c59e95-4001-11e9-a7c8-005056a27540 spec: clusterIP: 10.100.200.169 externalTrafficPolicy: Cluster ports: - nodePort: 30971 port: 9000 protocol: TCP targetPort: 9000 selector: component: minio sessionAffinity: None type: NodePort status: loadBalancer: {}

After making the change, you can now see which port that Minio is available on externally.

cormac@pks-cli:~/Velero$ kubectl get svc -n velero NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE minio NodePort 10.100.200.169 <none> 9000:32166/TCP 59m

Now the Minio S3 object store is accessible externally. Point a browser at the IP address of a Kubernetes worker node, and tag on the port (in this case 32166) shown above and you should get access to the Minio object store interface. Now we need to change one of the configuration files for Velero/Minio so that the velero client can get access to it for displaying logs, etc. This file is config/minio/05-backupstoragelocation.yaml. Edit this file, and uncomment the last line for the publicUrl, and set it appropriately.

apiVersion: velero.io/v1 kind: BackupStorageLocation metadata: name: default namespace: velero spec: provider: aws objectStorage: bucket: velero config: region: minio s3ForcePathStyle: "true" s3Url: http://minio.velero.svc:9000 # Uncomment the following line and provide the value of an externally # available URL for downloading logs, running Velero describe, and more. publicUrl: http://10.27.51.187:32166

Reapply the configuration using the command kubectl apply -f config/minio/05-backupstoragelocation.yaml. Now you should be able to access the logs using commands such as velero backups logs backup-name.

3. PKS interop – Privileged Containers

This was another issue that stumped me for a while. With PKS, through Pivotal Operations Manager, you create plans for the types of K8s clusters that you deploy. One of the options in a plan is to ‘Enable Privileged Containers’. Since this always stated ‘use with caution’, I always left it disabled. However, with Velero, and specifically with the restic portion (snapshots), I hit a problem when this was disabled. On trying to create a backup, I encountered these events on the restic daemon set:

cormac@pks-cli:~/Velero/config/minio$ kubectl describe ds restic -n velero Name: restic Selector: name=restic . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 49m daemonset-controller Error creating: pods "restic-v567s" is forbidden: pod.Spec.SecurityContext.RunAsUser is forbidden Warning FailedCreate 32m daemonset-controller Error creating: pods "restic-xv5v7" is forbidden: pod.Spec.SecurityContext.RunAsUser is forbidden Warning FailedCreate 15m daemonset-controller Error creating: pods "restic-vx2cb" is forbidden: pod.Spec.SecurityContext.RunAsUser is forbidden

So I went back to the Pivotal Ops Manager, went to the PKS tile, edited the plan, and enabled the checkbox for both ‘Enable Privileged Containers’ and ‘Disable DenyEscalatingExec’. I had to re-apply the PKS configuration, but once this completed (and I didn’t have to do anything with the K8s cluster btw, it is all taken care of by PKS), the restic pods were created successfully. Here are the buttons as they appear in the PKS tile > Plan.

OK – we are now ready to take a backup.

4. First Backup (CouchBase StatefulSet)

For my first backup, I wanted to take a StatefulSet – in my case, a CouchBase deployment that had been scaled out to 3 replicas. This means 3 pods and 3 PVs. Velero uses annotations to identify components for backup. Here I added special annotation to the volumes in the pods so that they can be identified by the restic utility for snapshotting. Note that this application is in its own namespace called “couchbase”.

cormac@pks-cli:~/Velero$ kubectl -n couchbase annotate pod/couchbase-0 backup.velero.io/backup-volumes=couchbase-data pod/couchbase-0 annotated cormac@pks-cli:~/Velero$ kubectl -n couchbase annotate pod/couchbase-1 backup.velero.io/backup-volumes=couchbase-data pod/couchbase-1 annotated cormac@pks-cli:~/Velero$ kubectl -n couchbase annotate pod/couchbase-2 backup.velero.io/backup-volumes=couchbase-data pod/couchbase-2 annotated

Now I can run the backup:

cormac@pks-cli:~/Velero$ velero backup create couchbase Backup request "couchbase" submitted successfully. Run `velero backup describe couchbase` or `velero backup logs couchbase` for more details. cormac@pks-cli:~/Velero$ velero backup describe couchbase --details Name: couchbase Namespace: velero Labels: velero.io/storage-location=default Annotations: <none> Phase: InProgress Namespaces: Included: * Excluded: <none> Resources: Included: * Excluded: <none> Cluster-scoped: auto Label selector: <none> Storage Location: default Snapshot PVs: auto TTL: 720h0m0s Hooks: <none> Backup Format Version: 1 Started: <n/a> Completed: <n/a> Expiration: 2019-04-05 13:25:25 +0100 IST Validation errors: <none> Persistent Volumes: <none included> Restic Backups: New: couchbase/couchbase-0: couchbase-data

cormac@pks-cli:~/Velero/$ velero backup get NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR couchbase Completed 2019-03-06 13:24:27 +0000 GMT 29d default <none>

At this point, it is also interesting to login to the Minio portal and see what has been backed up. Here is an overview from the top-most level – velero/restic/couchbase.

5. First Restore (CouchBase StatefulSet)

To do a real restore, let’s delete the couchbase namespace which has all of my application and data.

cormac@pks-cli:~/Velero/tests$ kubectl delete ns couchbase namespace "couchbase" deleted

cormac@pks-cli:~/Velero/tests$ kubectl get ns NAME STATUS AGE default Active 15d kube-public Active 15d kube-system Active 15d pks-system Active 15d velero Active 16h

Let’s now try to restore the delete namespace and all of its contents (pods, PVs, etc). Some of the more observant among you will notice that the backup name I am using is “cb2”, and not “couchbase”. That is only because I did a few different backup tests, and cb2 is the one I am restoring. You will specify your own backup name here.

cormac@pks-cli:~/Velero/tests$ velero restore create --from-backup cb2 Restore request "cb2-20190307090808" submitted successfully. Run `velero restore describe cb2-20190307090808` or `velero restore logs cb2-20190307090808` for more details.

cormac@pks-cli:~/Velero/tests$ velero restore describe cb2-20190307090808 Name: cb2-20190307090808 Namespace: velero Labels: <none> Annotations: <none> Backup: cb2 Namespaces: Included: * Excluded: <none> Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.ark.heptio.com, backups.velero.io, restores.ark.heptio.com, restores.velero.io Cluster-scoped: auto Namespace mappings: <none> Label selector: <none> Restore PVs: auto Phase: InProgress Validation errors: <none> Warnings: <none> Errors: <none>

After a few moment, the restore completes. I can also see that all 3 of my PVs are referenced. Next step is to verify that the application has indeed recovered and is usable.

cormac@pks-cli:~/Velero/tests$ velero restore describe cb2-20190307090808 --details Name: cb2-20190307090808 Namespace: velero Labels: <none> Annotations: <none> Backup: cb2 Namespaces: Included: * Excluded: <none> Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.ark.heptio.com, backups.velero.io, restores.ark.heptio.com, restores.velero.io Cluster-scoped: auto Namespace mappings: <none> Label selector: <none> Restore PVs: auto Phase: Completed Validation errors: <none> Warnings: <none> Errors: <none> Restic Restores: Completed: couchbase/couchbase-0: couchbase-data couchbase/couchbase-1: couchbase-data couchbase/couchbase-2: couchbase-data

Every thing appears to have come back successfully.

cormac@pks-cli:~/Velero/tests$ kubectl get ns NAME STATUS AGE couchbase Active 70s default Active 15d kube-public Active 15d kube-system Active 15d pks-system Active 15d velero Active 16h cormac@pks-cli:~/Velero/tests$ kubectl get pods -n couchbase NAME READY STATUS RESTARTS AGE couchbase-0 1/1 Running 0 90s couchbase-1 1/1 Running 0 90s couchbase-2 1/1 Running 0 90s cormac@pks-cli:~/Velero/tests$ kubectl get pv -n couchbase NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-8d676074-4032-11e9-842d-005056a27540 10Gi RWO Delete Bound velero/minio-pv-claim-1 minio-sc 16h pvc-9d9c537b-40b8-11e9-842d-005056a27540 1Gi RWO Delete Bound couchbase/couchbase-data-couchbase-0 couchbasesc 95s pvc-9d9d7846-40b8-11e9-842d-005056a27540 1Gi RWO Delete Bound couchbase/couchbase-data-couchbase-1 couchbasesc 95s pvc-9d9f60b9-40b8-11e9-842d-005056a27540 1Gi RWO Delete Bound couchbase/couchbase-data-couchbase-2 couchbasesc 91s cormac@pks-cli:~/Velero/tests$ kubectl get svc -n couchbase NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE couchbase ClusterIP None <none> 8091/TCP 117s couchbase-ui NodePort 10.100.200.80 <pending> 8091:31365/TCP 117s

I was also able to successfully connect to my CouchBase UI and examine the contents – all looked good to me. To close, I’d like to give a bit shout-out to my colleague Myles Gray, who fielded a lot of my K8s related questions. Cheer Myles!

[Update] OK, so I am not sure why it worked for me when I did my initial blog, but further attempts to restore this configuration resulted in CouchBase not starting. The reason for this seems to be that the IP addresses allocated to the CouchBase pods are hard-coded in either /opt/couchbase/var/lib/couchbase/ip or /opt/couchbase/var/lib/couchbase/ip_start on the pods. To get CouchBase back up and running, I had to replace the old entries (which were the IP addresses of the nodes which were backed up) with the new IP addresses of the restored pods. I ssh’ed onto the pods to do this. You can get the new IP address from /etc/hosts on each pod. Once the new entries were successfully updated, CouchBase restarted. However, it did not retain the original configuration, and seems to have reset to a default deployment. I’ll continue to reasearch into what else may need to be changed to bring the original config back.