Last update: April 29, 2020

Software-defined storage is not something new. One of the most popular is Ceph. I started with Ceph five years ago because I was looking into unified storage for OpenStack. There are many other solutions, but I like the Ceph because it is all in one solution for the block, object and file storage, and it is opensource. Inktank the company behind Ceph is later acquired by RedHat, but that made things even better. If you already have Ceph cluster running, it is easy to make use of it for Kubernetes. However, if you are designing entirely new on-premises Kubernetes cluster, you can run Ceph on top of it, and still, use it for other resources running on Kubernetes. This is where Rook comes into place. It provides deep Kubernetes integration made for cloud-native environments.

I'm excited about Rook. Not only because it solves persistent storage problems for Kubernetes, but also because it uses Ceph in the background. I designed at least five production grade Ceph clusters, so I'm pretty familiar with Ceph. For weeks I've been looking to write a post about Rook, and I finally made it.

Thanks @rook_io!



If you didn't hear about Rook.io yet, it is a Ceph on Kubernetes. In short, a cloud-native storage service. pic.twitter.com/6oeuGKHPzb — Alen Komljen (@alenkomljen) December 11, 2017

The Rook Way of Ceph deployment

The good news, you can run Ceph on Kubernetes and then use that storage for other Kubernetes resources. Rook, in a nutshell, is an operator which means that Rook manages Ceph cluster for you. To learn more about operators, a few weeks ago I wrote about Elasticsearch operator and how it works, so you might take a look if you want to dig deeper. Rook architecture diagram:

Of course, because Ceph requires extra drives to store the data, you would need to have a set of dedicated Kubernetes nodes. Currently, Rook is in an alpha state, but I'm expecting it to be production ready soon.

The easiest way to install Rook is using Helm. If you still didn't try Helm, it is the right time to do that. Add the new Helm repo and install Rook operator in kube-system namespace:

⚡ helm repo add rook-master https://charts.rook.io/master ⚡ helm search rook NAME CHART VERSION APP VERSION DESCRIPTION rook-master/rook v0.7.0-10.g3bcee98 File, Block, and Object Storage Services for yo... ⚡ helm install --name rook rook-master/rook \ --namespace kube-system \ --version v0.7.0-10.g3bcee98 \ --set rbacEnable=false

This Helm chart installs Rook operator and agents on each node. Check if everything is running and ready:

⚡ kubectl -n kube-system get pods -l 'app in (rook-operator, rook-agent)' NAME READY STATUS RESTARTS AGE rook-agent-4rhwt 1/1 Running 0 4m rook-agent-6s9v8 1/1 Running 0 4m rook-agent-8kgr9 1/1 Running 0 4m rook-agent-wqg9l 1/1 Running 0 4m rook-operator-845b8b8d4-p6cln 1/1 Running 0 4m

NOTE: If you are installing Rook on Kubernetes nodes running CoreOS or RancherOS you need to configure flexible volume first!

With Rook operator in place we have the new custom resources available. However, we still don't have Ceph cluster running.

To better understand Rook, first, you need to understand Ceph. Ceph is all in one solution for the block, object and file storage. The block storage (think of EBS) is what will probably be more interesting to you. Each time you create a Kubernetes Persistent Volume Claim or PVC, the Ceph will create the new volume. The main component responsible for block storage is Ceph OSD along with Ceph MON which provides cluster membership, configuration, and state. Those two components are enough to have a distributed block storage. There are other daemons for additional storage types and some helpers like API, etc.

Object storage (think of S3) is another layer, and Ceph component responsible for it is Ceph RadosGW. If you want to learn more about Ceph check official architecture docs.

Each Ceph OSD daemon handles only one physical drive. OSD stores the data in small objects which are part of placements groups or PGs. The placement groups are part of one pool which is distributed across other OSD nodes. Of course, you can have many pools, and each pool has defined number of replicas. Which means when you create a PVC, the data is everywhere, on each storage node and replicated.

For HA Ceph cluster you need at least three nodes. It is advisable to run an odd-number of monitors to have a quorum and default is set to three. Let's define the new Ceph cluster in rook namespace:

⚡ kubectl create namespace rook ⚡ cat <<EOF | kubectl create -n rook -f - apiVersion: rook.io/v1alpha1 kind: Cluster metadata: name: rook spec: dataDirHostPath: /var/lib/rook storage: useAllNodes: true useAllDevices: false storeConfig: storeType: bluestore databaseSizeMB: 1024 journalSizeMB: 1024 EOF

Please check the docs for all available options and explanation for above config. Wait a few minutes and Ceph cluster should be up and running:

⚡ kubectl get pods -n rook NAME READY STATUS RESTARTS AGE rook-api-854ffcf7b-6hnmw 1/1 Running 0 15m rook-ceph-mgr0-7957dc8d6c-xndkn 1/1 Running 0 15m rook-ceph-mon0-x6782 1/1 Running 0 16m rook-ceph-mon1-262tl 1/1 Running 0 16m rook-ceph-mon2-v2xv8 1/1 Running 0 16m rook-ceph-osd-6jfmh 1/1 Running 0 15m rook-ceph-osd-9f7w2 1/1 Running 0 15m rook-ceph-osd-ds4h7 1/1 Running 1 15m rook-ceph-osd-hkx87 1/1 Running 0 15m

For an experienced Ceph user, you want to be able to run ceph commands to check your cluster state. The easiest way is to deploy a separate rook-toolbox Pod and run commands from there:

⚡ cat <<EOF | kubectl create -n rook -f - apiVersion: v1 kind: Pod metadata: name: rook-tools namespace: rook spec: dnsPolicy: ClusterFirstWithHostNet containers: - name: rook-tools image: rook/toolbox:master imagePullPolicy: IfNotPresent env: - name: ROOK_ADMIN_SECRET valueFrom: secretKeyRef: name: rook-ceph-mon key: admin-secret securityContext: privileged: true volumeMounts: - mountPath: /dev name: dev - mountPath: /sys/bus name: sysbus - mountPath: /lib/modules name: libmodules - name: mon-endpoint-volume mountPath: /etc/rook hostNetwork: false volumes: - name: dev hostPath: path: /dev - name: sysbus hostPath: path: /sys/bus - name: libmodules hostPath: path: /lib/modules - name: mon-endpoint-volume configMap: name: rook-ceph-mon-endpoints items: - key: data path: mon-endpoints EOF

Now, for example, you can run a Ceph status check command:

⚡ kubectl -n rook exec rook-tools -- ceph -s cluster: id: 053cd70f-9b43-4854-862e-5bed29f1060d health: HEALTH_OK services: mon: 3 daemons, quorum rook-ceph-mon1,rook-ceph-mon0,rook-ceph-mon2 mgr: rook-ceph-mgr0(active) osd: 4 osds: 4 up, 4 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 8199 MB used, 145 GB / 153 GB avail pgs:

Ceph should report HEALTH_OK status, but we have 0 pools available. Before we can consume this cluster, we need to create at least one pool with the desired number of replicas. The number of replicas is usually set to three:

⚡ cat <<EOF | kubectl create -n rook -f - apiVersion: rook.io/v1alpha1 kind: Pool metadata: name: replicapool spec: replicated: size: 3 EOF

So finally, it is time to define the StorageClass for the above pool. Then we can create new PVCs:

⚡ cat <<EOF | kubectl create -f - apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-block provisioner: rook.io/block parameters: pool: replicapool EOF

Let's create a simple PVC to test if Ceph cluster is working fine:

⚡ cat <<EOF | kubectl create -f - kind: PersistentVolumeClaim apiVersion: v1 metadata: name: myclaim spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: rook-block EOF ⚡ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE myclaim Bound pvc-5f162665-1fa5-11e8-9056-525400474652 8Gi RWO rook-block 3s

There are many options for how to configure a Ceph cluster. Sometimes you have mixed drive types, and you want to have different pools for them. For example, fast storage with SSDs and slow with HDDs. Also, you may want to tune the Ceph cluster a little bit, but all those are advanced features. You should learn more about Ceph before moving forward with Rook.

Summary

A few weeks ago Rook became CNCF project which is good news. Keep in mind that Rook is not production ready yet, and some things can change. Can't wait to put it in place someday for large on-premises distributed storage. For any questions or concerns, please leave a comment. Stay tuned for the next one.