Last update: January 15, 2019

I wrote about Rook storage a few weeks ago, but maybe you already have Ceph cluster running in your datacenter. Or you prefer to use Ceph on separate nodes and without Kubernetes. Also, currently Rook is alpha software and not ready for production use. I would assume that this large Ceph cluster if you have one, is also used for other services outside Kubernetes. Whatever is the case it is simple to connect Ceph and Kubernetes together to provision persistent volumes on Kubernetes.

Previous blog post

Just blogged: Easy Way to Resize Kuberntes Persistent Volumes https://t.co/jjjLtpew7B #blogpost — Alen Komljen (@alenkomljen) March 12, 2018

Connect Ceph and Kubernetes

RBD client is used for interaction between Kubernetes and Ceph. Unfortunately, it is not available in official kube-controller-manager container. You could change kube controller manager image to include RBD, but that is not recommended. Instead, I will use external storage plugin for Ceph. This will create a separate rbd-provisioner pod which has rbd installed. My Kubernetes test cluster is RBAC enabled. If not, you can only create Deployment resource and skip the rest. In that case, don't forget to delete service account from deployment definition. Let's create all resources for rbd-provisioner with RBAC in kube-system namespace:

⚡ cat <<EOF | kubectl create -n kube-system -f - kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: rbd-provisioner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] - apiGroups: [""] resources: ["services"] resourceNames: ["kube-dns","coredns"] verbs: ["list", "get"] - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: rbd-provisioner subjects: - kind: ServiceAccount name: rbd-provisioner namespace: kube-system roleRef: kind: ClusterRole name: rbd-provisioner apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: rbd-provisioner rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: rbd-provisioner roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: rbd-provisioner subjects: - kind: ServiceAccount name: rbd-provisioner namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: rbd-provisioner --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: rbd-provisioner spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: rbd-provisioner spec: containers: - name: rbd-provisioner image: "quay.io/external_storage/rbd-provisioner:latest" env: - name: PROVISIONER_NAME value: ceph.com/rbd serviceAccount: rbd-provisioner EOF

Please check that quay.io/external_storage/rbd-provisioner:latest image has the same Ceph version installed as your Ceph cluster. You can check it like this on any machine running docker:

⚡ docker pull quay.io/external_storage/rbd-provisioner:latest ⚡ docker history quay.io/external_storage/rbd-provisioner:latest | grep CEPH_VERSION <missing> 15 hours ago /bin/sh -c #(nop) ENV CEPH_VERSION=luminous 0B

Wait a few minutes for RBD volume provisioner to be up and running:

⚡ kubectl get pods -l app=rbd-provisioner -n kube-system NAME READY STATUS RESTARTS AGE rbd-provisioner-77d75fdc5b-mpbpn 1/1 Running 1 1m

RBD volume provisioner needs admin key from Ceph to provision storage. To get the admin key from Ceph cluster use this command:

sudo ceph --cluster ceph auth get-key client.admin

NOTE: Run all commands that start with sudo on Ceph MON node. Also I'm using Jewel version of Ceph and rbd-provisioner is based on Jewel as well.

Then add this key to Kubernetes secrets:

⚡ kubectl create secret generic ceph-secret \ --type="kubernetes.io/rbd" \ --from-literal=key='AQBwruNY/lEmCxAAKS7tzZHSforkUE85htnA/g==' \ --namespace=kube-system

I will also create a separate Ceph pool for Kubernetes and the new client key as this Ceph cluster has cephx authentication enabled:

sudo ceph --cluster ceph osd pool create kube 1024 1024 sudo ceph --cluster ceph auth get-or-create client.kube mon 'allow r' osd 'allow rwx pool=kube' sudo ceph --cluster ceph auth get-key client.kube

Add the new client secret for kube pool into Kubernetes secrets:

⚡ kubectl create secret generic ceph-secret-kube \ --type="kubernetes.io/rbd" \ --from-literal=key='AQC/c+dYsXNUNBAAMTEW1/WnzXdmDZIBhcw6ug==' \ --namespace=kube-system

When both secrets are present create the new storage class. Let's call it fast-rbd :

⚡ cat <<EOF | kubectl create -f - apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast-rbd provisioner: ceph.com/rbd parameters: monitors: <monitor-1-ip>:6789, <monitor-2-ip>:6789, <monitor-3-ip>:6789 adminId: admin adminSecretName: ceph-secret adminSecretNamespace: kube-system pool: kube userId: kube userSecretName: ceph-secret-kube userSecretNamespace: kube-system imageFormat: "2" imageFeatures: layering EOF

And the last step is to create a simple PVC to test RBD volume provisioner:

⚡ cat <<EOF | kubectl create -f - kind: PersistentVolumeClaim apiVersion: v1 metadata: name: myclaim spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: fast-rbd EOF

That's it, the new volume created on Ceph cluster:

⚡ kubectl get pvc myclaim NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE myclaim Bound pvc-11559e19-2541-11e8-94dc-525400474652 8Gi RWO fast-rbd 1h

For any troubleshooting, run pvc describe command on a particular PVC.

Summary

This was a simple how-to guide to help you to connect Ceph and Kubernetes together. RBD volume provisioner is simple to deploy, but either way, I might create a Helm chart later. Stay tuned for the next one.