The next step is to start modeling the logic tree for our reconciliation loop. The pre-generated code for our event handling ( handler.go ) is basically a giant switch statement on the type of event the Operator will listen for. In our case we care about new v1alpha1.PrometheusReplica events.

I went through several redesigns of this loop and the final iteration was a phase for calculating status, installing any components that weren't running, and updating any components that didn't match the desired state.

Status

First, gather the current cluster state for Pods, StatefulSets, Services and Deployments that the Operator owns. The main goal is to establish whether the specific PRO instance is new, being created or running normally. If the particular object has never been seen before, it is set to install. Along the way, we build the status object to be appended at the end of the loop.

My heuristic for determining whether the cluster was Running or Creating was to simply count the unique Pod states for a set of label queries. If one or more was Pending or Creating , I assume Pods were still being created. If their aren't enough resources or qualified Nodes, it may stay in this state forever.

This lazy heruistic has one problem, which is that the Thanos Pods don't have liveness and readiness checks, so that this state doesn't truly represent them being ready. For example, the Store pods require several minutes of start up time to scan and build indices before they can respond to requests. It would be great for Thanos to provide an endpoint that returned this status for a readiness probe.

Install

The Install phase of the loop can be more accurately be named "install or repair", since both are really the same action in Kubernetes, because you only need to declare your desired state. Our desired state is either to have everything running, or if something has drifed out of spec, to reset its desired state back to what we expect.

At first, the block of create functions looks simple, but there is a lot of complexity hiding behind the curtains. It works well to write a function for each resource (Deployment, etc) of your application that returns the complete object to be created. There are two uses for this: to create it and to compare the current state to the desired state in the update phase of the loop.

When writing these, I stumbled upon a few things that worked well. Since our PRO object is very simple by design, it needs to be very clear how a configuration value is used within each resource. I logged each parameter that the resource would use and what it meant. For example, highlyAvailable: true might set the replica count to 2 on a Deployment and change the value of a flag on the created Pods. Spending this extra time to include debug output helps while you're developing as well as helps an admin with debugging later on.

Since I had already deployed the entire stack manually once I figured out the architecture, I “simply” had to translate these manifests over to Go code. This was extremely time consuming and frustrating because you have to figure out all of the names Kubernetes uses internally like v1.PodAffinityTerm. I found the in-page search on the Kubernetes API to be the most efficient method. For some instances the only solution I found was to scour Github search for relevant strings within other Operators or parts of the Kubernetes code base.

Every now and again you have to try to parse what these types of errors mean:

pkg/stub/handler.go:137:23: cannot use "github.com/robszumski/prometheus-replica-operator/ vendor/k8s.io/apimachinery/pkg/apis/meta/v1".LabelSelector literal (type "github.com/robsz umski/prometheus-replica-operator/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".LabelSelect or) as type *"github.com/robszumski/prometheus-replica-operator/vendor/k8s.io/apimachinery /pkg/apis/meta/v1".LabelSelector in field value

The answer: some of the fields are passed by reference and some aren’t, and it’s inconsistent. I just ended up doing trial and error but you can go look at the source as well.

Another tip is around label queries. Any reasonably complex app will use a few of these, and typically need to remain in sync across resources. I recommend writing a function to return each unique label query and then use that in Service and Deployment. If you need to update it later on, you can change it in one place.

Just like the install step, I figured this was going to be pretty easy, since a deep equality check between our desired state and current state is just a few lines of code:

// Update the Prometheus StatefulSet if the spec is different than desired if !reflect.DeepEqual(ssPromExisting.Spec, ssPromDesired.Spec) { err = sdk.Update(ssPromDesired) ... }

Turns out it’s not as easy as you think, because the object in the cluster has a bunch of default values added, eg. Protocol in the Ports section, which I did not specify in the object that I built up in the Operator.

Ports: [ { Name: "http", HostPort: 0, ContainerPort: 10902, - Protocol: "TCP", + Protocol: "", HostIP: "", }, { Name: "grpc", HostPort: 0, ContainerPort: 10901, - Protocol: "TCP", + Protocol: "", HostIP: "", }, { Name: "cluster", HostPort: 0, ContainerPort: 10900, - Protocol: "TCP", + Protocol: "", HostIP: "", }, ],

I recommend that you implement debug logging with a wonderful library called godebug, which produces a pretty diff as seen above.

To be forward-compatible with new defaults that Kubernetes may add, we need to do some deeper introspection. You need to be specific in your comparison, which will require some custom functions. Because you’re diving deeper into the objects, you will also not detect all changes that may have made to the object, since you’re only comparing the container args or replica count.

My strategy was to check a few of the critical paremeters in each object. For example, on the Prometheus StatefulSet, I check:

the total number of containers, for when we add a new one in a later version

the labels as these are important for routing

the replica count

the container args where our duration and retention metrics are set

The code to do this ends up being pretty simple:

if !reflect.DeepEqual(ssPromExisting.ObjectMeta.Labels, ssPromDesired.ObjectMeta.Labels) { logrus.Debugf(" Prometheus does not contain the correct labels") logrus.Debugf(pretty.Compare(ssPromExisting.ObjectMeta.Labels, ssPromDesired.ObjectMeta.Labels)) return true, ssPromDesired } else { logrus.Debugf(" Prometheus contains the correct correct labels") }

Now that we can sucessfully query the status of our objects and install or repair them if needed, we have a functioning Operator!