This is a two part series. Part One: the driver. Part Two: the provisioner

Photo by Alexandre Debiève on Unsplash

In Part One I have talked about how to write a custom persistent volume (PV) driver based on FlexVolume’s specification, which enables one go ahead to create static PVs integrated with customised PV lifecycle.

In this Part I will take you through how to create a dynamic provisioner which will use the driver created in Part One and dynamically provision your PVs.

Managing static PVs is a manual process which requires a spreadsheet type of work to keep track of PV inventory. This approach has a lot of overheads however it does serve some useful purposes. For example, it allows you to create an easy backup and disaster recovery (DR) plan because all the PVs are known and labelled logically based on use case. It could also serve as a underlying quota by only providing a limited inventory to cluster users.

Despite of the merits of static PV’s, in many other situations using a dynamic provisioner is more common since modern applications (especially in microservices environment) are mostly stateless. The stateful data are stored in cache or database, or it could be reinstated onto the PVs easily when the pod restarts.

We will utilise external-storage library(written in Go) to create a dynamic provisioner. It provides a simple interface for building out-of-tree persistent volume provisioners. It basically is a custom Kubernetes controller using client-go library (To those of whom are interested, I will detail how to create your own custom Kubernetes controller in a different article).

There are a few steps involve:

implement two external-storage’s interface

sort out dependancies

containerise the binary and run your deployment manifests

The Provisioner

There are a few key things you need to do [1]:

Import “github.com/kubernetes-incubator/external-storage/lib/controller” Implement two interfaces for your provisioner:

Provision(VolumeOptions) (*v1.PersistentVolume, error) Delete(*v1.PersistentVolume) error

Basically your provisioner will be observing the lifecycle of PVCs and conduct provisioning and deletion upon corresponding events occur.

The Provision method handles all your business logic that required and return a corresponding PV (similar to spec requires for the PV manifest). The business logic will include, for example, provisioning the physical persistent volume so the manual administration can be avoided.

The delete method will delete the physical persistent volume based on what PV’s retaining policy specified.

If you are planning to run multiple different instance of the provisioner that shares the same storage class, you will need to give each provisioner an identity, add the identity to returned PV via annotation and check it upon PV deletion. The caveat is that if you delete one of the provisioners, the PVs created by that provisioner will be parentless and won’t be handled by the rest of the provisioners due to unmatched identity.

Below are the sample codes (NOT a complete implementation) to help you to get started. One is the main, the other is the provisioner.

A Word on Go’s Package Management

There isn’t a universal package management tool for Go like in other languages despite of dep being the official experiment. There are other package management tools [2] being used for many applications for historical reasons. Such as Glide and Godep. Now there is a new proposal for Vgo but it’s still unclear when a common package management tool will be officially instated [Update: Seems like Vgo is approved, but the community is divided on its approach][Update 2: Consider using go modules instead] . Those tools are not compatible to each other in spite of Dep’s efforts trying to provide a common interface so the old repos with different package management tools can still be imported. However it still has problems and not always work with older repos that use different package management tools.

The Dependancies

external-storage uses glide as package management tool. Hence your new repo will need to use glide as well.

To bootstrap glide, simply run:

$ glide init

The command will generate a glide.yaml file which looks like below except the version . You would want to specify the external-storage release version so you can make sure your provisioner can run on the right version of Kubernetes.

package: github.com/your-github-username/awesome-repo

import:

- package: github.com/golang/glog

- package: github.com/kubernetes-incubator/external-storage

subpackages:

- lib/controller

- lib/util

version: v1.10.beta

- package: k8s.io/api

subpackages:

- core/v1

- package: k8s.io/apimachinery

subpackages:

- pkg/api/resource

- pkg/apis/meta/v1

- pkg/util/wait

- package: k8s.io/client-go

subpackages:

- kubernetes

- rest

After the glide.yaml file is ready, you need to run below command to install the dependancies to the vendor folder so the codes can be compiled rather than relying on libraries in $GOPATH :

$ glide install -v

NOTE: you must use -v flag so the nested vendor folders will be removed by glide, otherwise your compilation will fail.

Compilation

You can compile your code using just normal go build command. However you will need to use a base container that has go installed when creating a docker image.

The alternative is to compile a statically linked binary, for example to compile for linux:

CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo ldflags '-w' -o awesome-provisioner

The benefit of doing so is that you can use any linux based images including scratch .

To Finish

After you have containerised your provisioner, the one thing left to do is to write a deployment manifest then simply deploy it to your Kubernetes cluster. Note you will need to have relevant secret created either by your provisioner or referenced if you are passing secret to the FlexVolume .

Reference

[1] Write a Out-Of-Tree Dynamic Provisioner

[1] Go Package Management Tools