Last update: June 14, 2018

The previous post about Rook got great attention from the community. But, I also got a lot of questions about persistence on Kubernetes in general. Among many questions, there were a few about resizing persistent volume claims or PVCs. If you are using on-premises storage solutions like Ceph (Rook) this is not the big issue. The part of the reason is that when you create a volume in Ceph, let's say a 100GB, this space is not actually allocated. But, if you are using AWS you will pay for that unused space even if you don't need it at the moment. Why is this so hard? The storage is elastic in the cloud, so resize is easy, but the problem is that resize should happen on the file system level also. Manual resize is possible, but it isn't convenient.

Previous blog post

Just blogged: @rook_io a Cloud Native On-Premises Persistent Storage for Kubernetes on Kubernetes https://t.co/Hii4V1tery #blogpost — Alen Komljen (@alenkomljen) March 4, 2018

Kubernetes Community to the Rescue

It seems that it would be great if we could grow persistent volume claims in size when needed. Actually, this is available from Kubernetes v1.8 and it is an alpha feature. In v1.9 resize is possible with those storage provisioners:

gcePersistentDisk

awsElasticBlockStore

OpenStack Cinder

glusterfs

rbd

Unfortunately, this is still not available for Rook. To be able to use this feature you need to allow expanding PVCs by setting ExpandPersistentVolumes feature gate to true. Also, enable PersistentVolumeClaimResize admission plugin if you want to prevent resizing of all claims unless you enable resize in storage class. How to update feature gates and admission plugins depends on how you deployed Kubernetes cluster, but in general, add the following parameters to kube-apiserver process --feature-gates=ExpandPersistentVolumes=true,... and --admission-control=PersistentVolumeClaimResize,... , and also add the same feature gate to kube-controller-manager .

For the purpose of this blog post, I deployed a Kubernetes cluster using Heptio AWS quickstart guide which deploys Kubernetes with kubeadm. Instructions in this blog post should work for any cluster deployed with kubeadm v1.9. After the stack is up and running you need to update manifests files kube-apiserver.yaml and kube-controller-manager.yaml :

sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml spec: containers: - command: - kube-apiserver - --admission-control=PersistentVolumeClaimResize,Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota - --feature-gates=ExpandPersistentVolumes=true sudo vim /etc/kubernetes/manifests/kube-controller-manager.yaml spec: containers: - command: - kube-controller-manager - --feature-gates=ExpandPersistentVolumes=true

Then you need to restart kubelet to apply those changes:

sudo systemctl restart kubelet

After enabling changed configurations you need to set allowVolumeExpansion for storage class if you want to enable expansion. I will create the new gp2-resize storage class:

⚡ cat <<EOF | kubectl create -f - apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp2-resize provisioner: kubernetes.io/aws-ebs parameters: type: gp2 allowVolumeExpansion: true EOF

And let's create the simple PVC:

⚡ cat <<EOF | kubectl create -f - kind: PersistentVolumeClaim apiVersion: v1 metadata: name: myclaim spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: gp2-resize EOF ⚡ kubectl get pvc myclaim NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE myclaim Bound pvc-d15808da-2575-11e8-a133-02cfcfdc585a 8Gi RWO gp2-resize 36s

In this case, PVC is not used by any Pod. I can go ahead and make it larger, from 8GB to 12GB:

⚡ kubectl edit pvc myclaim spec: resources: requests: storage: 12Gi

This will trigger the expansion of this PVC. Supported file systems are ext3, ext4, and XFS. Please note that file system resize will not happen unless you recreate your Pods. This is valid if the PVC is in use.

NOTE: Online resize support should be available in Kubernetes v1.11, and you won't need to recreate resources for file system resize.

Wait a few minutes and check for the updated EBS size:

⚡ aws --region $REGION ec2 describe-volumes --filters Name=tag-key,Values="kubernetes.io/created-for/pvc/name" Name=tag-value,Values="myclaim" --query 'Volumes[0].Size' 12

However, PVC size in Kubernetes was not updated to reflect the new size:

⚡ kubectl get pvc myclaim -o=jsonpath="{.status.capacity.storage}" 8Gi

Maybe I'm missing something or this is a bug. After resize you will have to wait at least 6 hours before another modification per EBS volume.

Summary

With features like online volumes resize, stateful apps will be easier to use on Kubernetes. While not that straightforward to use at the moment, I'm sure that resize will become better over time. After all, this is an alpha feature. Stay tuned for the next one.