The Kubernetes watch API in-depth

A key piece of this entire project was being able to act on real-time information from the Kubernetes API. I didn’t want to have to rely on a polling mechanism to query for changes from the cluster, but rather receive the changes that happen directly through a stream.

The Node.js backend utilizes a Kubernetes client library written by engineers at Godaddy. This particular library makes it easy to utilize the watch flag that you can pass to the Kubernetes API to keep a persistent connection open (via HTTP or websocket) and stream in changes to resources in real-time.

Here is an example of how this code works:

Different events that come from the K8s watch API — full code available at https://github.com/thenayr/kubernetes-vr-api/blob/master/index.js

I did run into a few caveats in dealing with the watch API. Unbeknownst to me at the time, the Kuberenetes API will actually assign a timeout to your connection regardless of what you set on the client side. I made the assumption that the watch request would persist until I killed it on the client side. You can read a bit more about how to deal with that bit here: http://stackoverflow.com/questions/33480560/kubernetes-api-server-drops-watch-connections

Determining when a new pod was created AND ready

Another “gotcha” to mention with the way the watch responses come through deals with the events ADDED and MODIFIED. When I was debugging an earlier version of the project, I noticed that pods that come through on the ADDED event were missing a lot of the metadata that I needed to pull in (IP address, labels, etc). It turns out that ADDED is the first event that fires for a new pod, even before that pod actually enters any sort of ready state.

Even more troublesome, creating a single new pod would fire off a series of events like so:

ADDED

MODIFIED

MODIFIED

MODIFIED

MODIFIED

Each phase that the pod went through after first being added would send another event to the stream. Given that my goal was simply to take a newly created pod and it’s metadata and insert it into my A-Frame scene, this made it rather difficult to distinguish when a new pod was actually created and ready, especially when multiple pod creations and deletions were happening simultaneously. I eventually found the right combination of events to determine when that state occurs:

Events to use to find newly added ready pods. Full code available here — https://github.com/thenayr/kubernetes-vr-api/blob/master/index.js

That is to say, I couldn’t use the ADDED event in the stream to generate new objects in my scene, instead I had to rely on a combination of the MODIFIED event and a series of metadata objects that show up in that event. It would be nice to have a READY event added to the API that would in effect do the same thing.

Placing Pods into a A-frame VR scene

An earlier prototype of the scene with some placeholder pods

Now that I had the event stream properly detecting pods upon creation and deletion, I had to come up with a way to get them to render directly into my VR scene in real-time.

This is where React really shines, I was able to build a Pod component which I could then have dynamically added or removed from the scene whenever the appropriate event was triggered. The following is a simplified gist of what that component looks like (I stripped out some of the dynamic texturing and extraneous functionality)

I then built a parent component to handle the actual spawning of the child Pods which would handle the positioning, retrieval of metadata via web sockets and overall lifecycle of the child pod.

There is quite a lot of code here that handles communication via websockets to and from the backend Node.js API. Important to note is the initial provisioning that happens via the loadPodsFromServer function. This emits a websocket event to the Node application which in turn responds with a JSON blob including my existing pods.

I wanted the pods to spawn in random positions in the scene and fall out of the sky. The function randomPosition handles that by setting the X,Y,Z coordinates each time a new pod component is going to be created.

Later on in the development cycle I decided I wanted to texturize the boxes depending on the type of image the pod was actually running. To implement this I added a simple labeling system to my (example) Kubernetes manifest files.

spec:

replicas: x

revisionHistoryLimit: x

template:

metadata:

labels:

type: nginx

When the Node API emits an event over web sockets, it includes this metadata label type and uses it to map back to a texture image in the front-end application:

The Node API side of the code checks to see if a pod has this label assigned, otherwise gives it a default pod texture (blue with the Kubernetes logo):

Inside of a separate React component I define my A-frame assets using the asset management system :

This entire system is very rudimentary and rigid for the most part (it has no notion of textures for types that don’t exist), but it got the job done for the limited subset of pods I was dealing with for my demonstration.

Here is the (browser and keyboard) result of what these texture mappings look like: