With the v0.3 release, Rook brings cloud-native storage into the world of Kubernetes, running itself as a Kubernetes application. Block storage, object storage, and shared file systems can now be natively integrated with your Kubernetes applications.

Rook manages the storage cluster with the Operator pattern. An operator automates management that traditionally might have been done by a cluster admin. Whatever operational knowledge a cluster admin would need to setup and monitor the system, the operator will automate. The operator also acts as a Kubernetes extension by implementing third party resources (TPRs).

The operator is started with a simple Kubernetes deployment. Create a file rook-operator.yaml:

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: rook-operator

spec:

replicas: 1

template:

metadata:

labels:

name: rook-operator

spec:

containers:

- name: rook-operator

image: quay.io/rook/rook-operator

env:

- name: ROOK_OPERATOR_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

Start the operator.

kubectl create -f rook-operator.yaml

Once the operator is running, you’ll be able to create your first Rook cluster with this yaml: rook-cluster.yaml

apiVersion: rook.io/v1beta1

kind: Cluster

metadata:

name: my-rook

spec:

namespace: rook

version: latest

useAllDevices: false

Create the cluster.

kubectl create -f rook-cluster.yaml

That’s it. In a matter of minutes you will have storage available to your Kubernetes applications.

The operator’s job is to ensure you have a healthy storage cluster. On the first run, all the necessary Kubernetes primitives will be created, after which the operator will continuously monitor the storage components.

Consuming the Storage

Volume Claim

To consume Rook block storage, your applications will first need to create a storage class based on the Ceph RBD volume plugin: rook-storageclass.yaml

apiVersion: storage.k8s.io/v1beta1

kind: StorageClass

metadata:

name: rook-block

namespace: rook

provisioner: kubernetes.io/rbd

parameters:

monitors: INSERT_HERE

adminId: admin

adminSecretName: rook-admin

adminSecretNamespace: rook

pool: rook

userId: rook-rbd-user

userSecretName: rook-rbd-user

You may have noticed that you need to insert the monitor endpoints in the yaml. This is a rather painful process that we are looking to simplify in our next release with a Rook volume plugin for Kubernetes. For now, you will need to run a command to extract the endpoints:

export MONS=$(kubectl -n rook get pod mon0 mon1 mon2 -o json|jq ".items[].status.podIP"|tr -d "\""|sed -e 's/$/:6790/'|paste -s -d, -)

And finally you can replace the endpoints in the yaml and create the storage class.

sed 's#INSERT_HERE#'$MONS'#' rook-storageclass.yaml | kubectl create -f -

Now to consume the storage you will create a volume claim in your application. For example:

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

name: mysql-pv-claim

annotations:

volume.beta.kubernetes.io/storage-class: rook-block

labels:

app: mysql

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 20Gi

Then your container spec might use the volume claim like this:

spec:

containers:

- image: mysql:5.6

name: mysql

volumeMounts:

- name: mysql-persistent-storage

mountPath: /var/lib/mysql

volumes:

- name: mysql-persistent-storage

persistentVolumeClaim:

claimName: mysql-pv-claim

This example was extracted from the yaml here and is described in more detail in the full walkthrough.

Rook Client

Another way to connect to the Rook storage is through the Rook client container. This provides a simple environment for testing the block, object, and file storage directly. While this isn’t how your Kubernetes applications will likely consume the storage, it is a playground to experiment with Rook until we have more complete integration with Kubernetes volume plugins.

To run the tool, all you need to do is start and connect to the client tools pod and you’ll have an environment to execute rook commands to manage your cluster.

Create the client pod: rook-client.yaml

apiVersion: v1

kind: Pod

metadata:

name: rook-client

namespace: rook

spec:

containers:

- name: rook-client

image: quay.io/rook/rook-client:latest

imagePullPolicy: IfNotPresent

command: ["sleep", "36500d"]

securityContext:

privileged: true

volumeMounts:

- mountPath: /dev

name: dev

- mountPath: /sys

name: sys

- mountPath: /lib/modules

name: libmodules

volumes:

- name: dev

hostPath:

path: /dev

- name: sys

hostPath:

path: /sys

- name: libmodules

hostPath:

path: /lib/modules

Start the client pod.

kubectl create -f rook-client.yml

Wait for the pod to start, then connect:

kubectl exec -it rook-client bash

The tool automatically picks up on the settings to connect to the management API and you’re all set to manage your Rook cluster.

See the nodes in the cluster:

rook node ls

See the cluster status:

rook status

Create an S3 object store:

rook object create

For the full usage, see the readme or just ask for help:

rook --help

Kubernetes Resource Design

The operator will automatically initialize the cluster by creating several Kubernetes-native resources and then monitoring the resources to ensure cluster health.

Secrets

At first run, the cephx admin and monitor secrets are generated and stored in Kubernetes secrets.

Services

Pods are started in the cluster, one for each of three Ceph monitors. These pods are critical to the health of the storage cluster.

Where there are at least three nodes in the cluster, anti-affinity is specified so the failure domain will be across nodes.

After the pods are started, the operator waits until Ceph monitor quorum is established before continuing with the OSDs.

The operator starts a Go routine to watch the health of the monitor pods.

DaemonSet

A daemon set is started for Ceph OSDs on all nodes labeled for storage.

As the OSD pod starts on each machine, an EmptyDir is created to store configuration. By default, data will also be stored in this directory. In the very near future we will give the cluster admin the ability to specify devices or directories that can be used locally for storage on the host.

The pod will then will watch the health of the OSD locally.

Third Party Resources

For the cluster admin to make changes to the cluster, Rook provides a simple TPR to specify how the storage cluster should be configured. The operator will watch the TPRs and apply the changes to the cluster. In the near future, TPRs will also be included to enable S3/Swift compatible Object storage (RGW), shared file system (MDS), managing of cephx users (to connect with the RBD volume plugin), and more.

Management API

A Rook API deployment is started to provide a RESTful management plane and simplify some of the management tasks around the Rook cluster. The API enables retrieving status and updating configuration of the cluster. The primary consumer of the API is the rook client tool.

Reliability

If the operator stops running for any reason, the storage cluster will continue to run as expected. Ceph Mons, OSDs, MDS, and RGW services are agnostic that the operator even exists. Basic monitoring of the cluster health would be done by Kubernetes itself at a basic level. When the operator is started again, the more complete monitoring will resume and changes made to desired state in the third party resources would be applied at that point.

Takeaway

The Rook operator’s goal is to automate storage in your cluster, keep the storage running reliably, and keep your data safe. The operator is just starting down this path. We look forward to your feedback and contributions in this journey to version 1.0 and beyond!

For more documentation and sample yaml files, see the Rook github.