Discussion on Reddit/Clojure

Also take a look at sunng’s library Stavka which can read and watch Kubernetes configMaps, and via the same mechanism, could trivially watch a pods labels and annotations, as described in this article.

Introduction

Feature flags are used to change runtime behavior of a program without restarting it. While they are essential in a cloud native environment, the must be used judiciously. In the past, they could be fairly tricky to implement across an organization’s microservices, but Kubernetes has made them trivial to implement. Here we’re going to implement them via labels and annotations, but you can also implement them by connecting to the Kubernetes API directly.

In Kubernetes, labels are part of a resources’ identity, and can be used via selectors to include/exclude particular resources. Annotations are similar, but do not participate in a resources’ identity, and cannot be used to select resources. Annotations are frequently used to store data about a resource.

Labels are specified in our yaml at spec.template.metadata.labels, and annotations right next door at spec.template.metadata.annotations. They will be available to our Clojure code at /etc/podinfo/labels and /etc/podinfo/annotations respectively.

Sample use cases

Turn on/off a repl in a specific instance.

Turn on/off profiling of a specific instance.

Change the logging level in production, to capture detailed logs during a specific event.

Change caching strategy at runtime.

Change timeouts in production.

Toggle spec verification.

Wrangling labels and annotations from the shell.

# Add a label $ kubectl label pod my-pod-name a-label = foo # Show labels $ kubectl get pods --show-labels # If you only want to show specific labels, use -L=<label1>,<label2> # Update a label $ kubectl label pod my-pod-name a-label = bar --override # Delete a label $ kubectl label pod my-pod-name a-label- # Add an annotation $ kubectl annotatate pod my-pod-name an-annotation = foo # Show annotations $ kubectl describe pod my-pod-name # Update an annotation $ kubectl annotation pod my-pod-name an-annotation = foo --override # Delete an annotation $ kubectl annotation pod my-pod-name an-annotation-

We’ll use the Kubernetes downward-api to expose labels and annotations directly to our application. We’ll end up with two files (“labels” and “annotations”) in /etc/podinfo.

First we add the downward api to spec.volumes. Note that we’re adding both labels and annotations into the same volume. You can also expose certain container fields as a file consisting of a single value, see here for more info.

volumes : - name : podinfo downwardAPI : items : - path : " labels" fieldRef : fieldPath : metadata.labels - path : " annotations" fieldRef : fieldPath : metadata.annotations

Now we’re going to specify where we mount it into the container. We’ll add the following to spec.template.volumeMounts.

volumeMounts : - name : podinfo mountPath : /etc/podinfo readOnly : false

Our deps are simple, and are saved as deps.edn locally.

{ deps { juxt/dirwatch { :mvn/version "0.2.3" }}}

Note: For juxt/dirwatch, use version 0.2.3, as there is a bug in 0.2.4 that stops it from working.

Here’s the script we’re going to run. (It’s saved locally as script.clj)

( require ' [ juxt.dirwatch :refer ( watch-dir )]) ( require ' [ clojure.java.io :as io ]) ( require ' [ clojure.edn :as edn ]) ( def annotations ( atom {})) ;; This is just so we can see changes reflected in the log. ( add-watch annotations :change ( fn [ ctx k old-value new-value ] ( when ( not= old-value new-value ) ( println new-value )))) ( defn load-props "The files produced by the downward api are close enough to property files that we'll use the built in property reader to parse them." [ file-handle ] ( with-open [ ^ java.io.Reader reader ( io/reader file-handle )] ( let [ props ( java.util.Properties. )] ( .load props reader ) ( ->> ( for [[ k v ] props ] [( keyword k ) ( edn/read-string v )]) ( into {} ))))) ( defn downward-api-watcher "When a file event comes in, reload the atom. Note that we don't use the given file handle, as it will point to the temporary file." [{ :keys [ file, count, action ]}] ( let [ fname ( .getName file )] ;; Race condition? ( when ( = fname "annotations" ) ( reset! annotations ( load-props ( io/file "/etc/podinfo/annotations" )))))) ( watch-dir downward-api-watcher ( io/file "/etc/podinfo/" ))

We install it into our cluster via:

kubectl run -it --restart = Never \ --image = clojure:openjdk-11-tools-deps \ --dry-run \ -oyaml \ --rm = true \ --command -- "clojure" "-Sdeps" " $( cat deps.edn ) " "-e" " $( cat script.clj ) "

;; Example events { :file # object [ java.io.File 0 x6c025238 /etc/podinfo/..data/labels ] , :count 1 , :action :create } { :file # object [ java.io.File 0 x26b1ce79 /etc/podinfo/..data/annotations ] , :count 1 , :action :create }

You can test it as following.

$ kubectl apply -f feature-flag.yaml deployment.extensions "feature-flag" created $ kubectl get pods NAME READY STATUS RESTARTS AGE feature-flag-78db4f4694-cxmrp 1/1 Running 0 6s $ kubectl annotate pod feature-flag-78db4f4694-cxmrp foo = bar pod "feature-flag-78db4f4694-cxmrp" annotated

Here is a yaml listing, with some cut down Clojure code, if you want to test it simply as a single resource.

apiVersion : extensions/v1beta1 kind : Deployment metadata : creationTimestamp : null labels : run : feature-flag name : feature-flag spec : replicas : 1 selector : matchLabels : run : feature-flag strategy : {} template : metadata : creationTimestamp : null labels : run : feature-flag annotations : - cloudnativeclojure.org/my_cool_feature="off" spec : containers : - command : - clojure - -Sdeps - ' {deps {juxt/dirwatch {:mvn/version "0.2.3"}}}' - -e - |- (require '[juxt.dirwatch :refer (watch-dir)]) (watch-dir println (clojure.java.io/file "/etc/podinfo/")) image: clojure:openjdk-11-tools-deps name: feature-flag resources: {} volumeMounts: - name: podinfo mountPath: /etc/podinfo readOnly: false volumes : - name : podinfo downwardAPI : items : - path : " labels" fieldRef : fieldPath : metadata.labels - path : " annotations" fieldRef : fieldPath : metadata.annotations status : {}

Thanks to Jonathan Morris, Mia Tyzzer, and Arthur Ulfeldt for reading early drafts of this.