A few months back, I wrote an article which got a bit of interest around the issues configuring and maintaining multiple clusters, and keeping the components required to make them useful in sync. Essentially, the missing piece of the puzzle was that there was no cluster aware configuration management tool.

Internally, we created an excellent tool at $work to solve this using jsonnet, which has been very nice because it’s meant we get to use actual code to solve this problem. The only issue is that the code we have to write in is relatively niche!

When Pulumi was announced back in June, I got very excited. After seeing the that Pulumi now supports Kubernetes natively, I wanted to dig in and see if it would help with the configuration management problem.

Introductory Concepts

Before I speak about the Kubernete–specific components, I want to do a brief introduction into what Pulumi actually is.

Pulumi is a tool which allows you to create cloud resources using real programming languages. In my personal opinion, it is essentially terraform with full programming languages on top of it, instead of HCL.

Having real programming languages means you can make the resources you configure as flexible or complex as you like. Each provider (AWS, GCE, Azure and Kubernetes) has support for different languages. Currently, the AWS provider has support for Go, Python and Javascript/Typescript whereas the Kubernetes provider has support only for Javascript/Typescript at the moment.

Stacks

Pulumi uses the concept of a stack to segregate things like environments or feature branches. For example, you may have a stack for your development environment and one for your production environment.

These stacks store their state in a similar fashion to terraform. You can either store the state in:

The public pulumi web backend

locally on the filesystem

There is also an enterprise hosted stack storage provider. Hopefully it’ll be possible to have an open source stack storage some time soon.

Components

Components allow you to share reusable modules, in a similar manner to terraform modules. This means you can write reusable code and create boilerplate for regularly reused resources, like S3 buckets.

Configuration

Finally, there’s configuration ability within Pulumi stacks and components. What this allows you to do is differentiation configuration in the code depending on the stack you’re using. You specify these configuration values like so:

pulumi config set name my_name

And then you can reference that within your code!

Kubernetes Configuration Management

If you remember back to my previous post, the issue I was trying to solve was being able to install components that every cluster needs (as an example, an ingress controller) to all the clusters but with often differing configuration values (for example, the path to a certificate arn in AWS ACM). Helm helps with this in that it allows you to specify values when installing, but then managing, maintaining and storing those values for each cluster becomes difficult, and applying them also becomes hard.

Pulumi

There are two main reasons Pulumi is helping here. Firstly, it allows you to differentiate kubernetes clusters within stacks. As an example, let’s say I have two clusters - one in GKE and one in Digital Ocean. Here you can see them in my kubernetes contexts:

I can initiate a stack for each of these clusters with Pulumi, like so:

# create a Pulumi.yml first! cat << EOF > Pulumi.yml name: nginx runtime: nodejs description: An example stack for Pulumi EOF pulumi stack init gcloud pulumi config set kubernetes:context [email protected]

Obviously, you’d repeat the stack and config steps for each cluster!

Now, if I want to actually deploy something to the stack, I need to write some code. If you’re not familiar with typescript (I wasn’t, until I wrote this) you’ll need to generate a package.json and a tsconfig.json

Fortunately, Pulumi automates this for us!

pulumi new --generate-only --force

If you’re doing a Kubernetes thing, you’ll need to select kubernetes-typescript from the template prompt.

Now we’re ready to finally write some code!

Deploying a standard Kubernetes resource

When you ran pulumi new , you got an index.ts file. In it, it imports pulumi:

import * as k8s from " @pulumi/kubernetes " ;

You can now write standard typescript and generate Kubernetes resources. Here’s an example nginx pod as a deployment:

import * as k8s from " @pulumi/kubernetes " ; // set some defaults const defaults = { name : " nginx " , namespace : " default " , labels : { app : " nginx " }, serviceSelector : { app : " nginx " }, }; // create the deployment const apacheDeployment = new k8s . apps . v1 . Deployment ( defaults . name , { metadata : { namespace : defaults . namespace , name : defaults . name , labels : defaults . labels }, spec : { replicas : 1 , selector : { matchLabels : defaults . labels , }, template : { metadata : { labels : defaults . labels }, spec : { containers : [ { name : defaults . name , image : `nginx:1.7.9` , ports : [ { name : " http " , containerPort : 80 , }, { name : " https " , containerPort : 443 , } ], } ], }, }, } });

Here you can already see the power that writing true code has - defining a constant for defaults and allowing us to use those values in the declaration of the resource means less duplicated code and less copy/pasting.

The real power comes when using the config options we set earlier. Assuming we have two Pulumi stacks, gcloud and digitalocean :

pulumi stack ls NAME LAST UPDATE RESOURCE COUNT digitalocean * n/a n/a gcloud n/a n/a

and these stacks are mapped to different contexts, like so:

pulumi config -s gcloud KEY VALUE kubernetes:context [email protected] pulumi config -s digitalocean KEY VALUE kubernetes:context digitalocean

you can now also set different configuration options and keys which can be used in the code.

pulumi config set imageTag 1.14-alpine -s gcloud pulumi config set imageTag 1.15-alpine -s digitalocean

This will write out these values into a Pulumi.<stackname>.yaml in the project directory:

cat Pulumi.digitalocean.yaml config: kubernetes:context: digitalocean pulumi-example:imageTag: 1.15-alpine

and we can now use this in the code very easily:

let config = new pulumi . Config ( " pulumi-example " ); // use the name field in the Pulumi.yaml here let imageTag = config . require ( " imageTag " ); // this is part of the pod spec, you'll need the rest of the code too! image : `nginx: ${ imageTag } ` ,

Now, use Pulumi from your project’s root to see what would happen:

pulumi up -s gcloud Previewing update of stack 'gcloud' Previewing changes: Type Name Plan Info + pulumi:pulumi:Stack pulumi-example-gcloud create + └─ kubernetes:apps:Deployment nginx create info: 2 changes previewed: + 2 resources to create Do you want to perform this update? yes Updating stack 'gcloud' Performing changes: Type Name Status Info + pulumi:pulumi:Stack pulumi-example-gcloud created + └─ kubernetes:apps:Deployment nginx created info: 2 changes performed: + 2 resources created Update duration: 17.704161467s Permalink: file:///Users/Lee/.pulumi/stacks/gcloud.json

Obviously you can specify whicever stack you need as required!

Let’s verify what happened…

kubectx digitalocean # switch to DO context kubectl get deployment -o wide NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR nginx 1 1 1 1 1m nginx nginx:1.15-alpine app = nginx kubectx [email protected] # let's look now at gcloud context Switched to context [email protected]" . k get deployment -o wide NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR nginx 1 1 1 1 3m nginx nginx:1.14-alpine app = nginx

Okay, so as you can see here, I’ve deployed an nginx deployment to two different clusters, with two different images with very little effort and energy. Awesome!

Going further - Helm

What makes this really really awesome is that Pulumi already supports Helm charts!

In my previous post, I made the comment that Helm has lots of community supported charts which have done a whole load of configuration for you. However, Helm suffers from 2 main problems (in my humble opinion)

Templating yaml with Go templates is extremely painful when doing complex tasks

The helm chart community can be slow to merge pull requests. This means if you find an issue with a chart, you unfortunately have to fork it, and host it yourself.

Pulumi really helps and solves this problem. Let me show you how!

First, let’s create a Pulumi component using a helm chart:

import * as k8s from " @pulumi/kubernetes " ; const redis = new k8s . helm . v2 . Chart ( " redis " , { repo : " stable " , chart : " redis " , version : " 3.10.0 " , values : { usePassword : true , rbac : { create : true }, } });

And now preview what would happen on this stack:

pulumi preview -s digitalocean Previewing update of stack 'digitalocean' Previewing changes: Type Name Plan Info + pulumi:pulumi:Stack pulumi-helm-example-digitalocean create + └─ kubernetes:helm.sh:Chart redis create + ├─ kubernetes:core:Secret redis create + ├─ kubernetes:rbac.authorization.k8s.io:RoleBinding redis create + ├─ kubernetes:core:Service redis-slave create + ├─ kubernetes:core:Service redis-master create + ├─ kubernetes:core:ConfigMap redis-health create + ├─ kubernetes:apps:StatefulSet redis-master create + └─ kubernetes:extensions:Deployment redis-slave create info: 9 changes previewed: + 9 resources to create

As you can see, we’re generating the resources automatically because Pulumi renders the helm chart for us, and then creates the resources, which really is very awesome.

However, it gets more awesome when you see there’s a callback called transformations . This allows you to patch and manipulate the generated resources! For example:

import * as k8s from " @pulumi/kubernetes " ; const redis = new k8s . helm . v2 . Chart ( " redis " , { repo : " stable " , chart : " redis " , version : " 3.10.0 " , values : { usePassword : true , rbac : { create : true }, }, transformations : [ // Make every service private to the cluster, i.e., turn all services into ClusterIP instead of // LoadBalancer. ( obj : any ) => { if ( obj . kind == " Service " && obj . apiVersion == " v1 " ) { if ( obj . spec && obj . spec . type && obj . spec . type == " LoadBalancer " ) { obj . spec . type = " ClusterIP " ; } } } ] });

Of course, you can combine this with the configuration options from before as well, so you can override these as needed.

I think it’s worth emphasising that this improves the implementation time for any Kubernetes deployed resource dramatically. Before, if you wanted to use something other than helm to deploy something, you had to write it from scratch. Pulumi’s ability to import and them manipulate rendered helm charts is a massive massive win for the operator and the community!

Wrap up

I think Pulumi is going to change the way we deploy things to Kubernetes, but it’s also definitely in the running to solve the configuration management problem. Writing resources in code is much much better than writing yaml or even jsonnet, and having the ability to be flexible with your deployments and manifests using the Pulumi concepts is really exciting.

I’ve put the code examples from the blog post in a github repo for people to look at and improve. I really hope people try out Pulumi!