Kubernetes is the hottest kid on the block among container orchestration tools right now. I started writing this post when we decided to go with Kubernetes at Twyla a year ago, and since then, the developments in the ecosystem have been simply overwhelming. In my opinion, the attention Kubernetes gets is completely deserved, due to the following reasons:

It is a complete solution that is based on a fundamental set of ideas. These ideas are explained in the Borg, Omega and Kubernetes article that compares the consecutive orchestration solutions developed at Google, and the lessons learned.

While it is container-native, Kubernetes is not limited to a single container platform, and the container platform is extended with e.g. networking and storage features.

It offers an open and well-designed API, in addition to various patterns that suit differing workflows. The wonderful thing is that there is a very well-governed community process whereby the API is constantly developed further. You have to spend effort keeping up, but regularly receive goodies in return.

In this tutorial, I want to document my journey of learning Kubernetes, clear up some points that tripped me as a beginner, and try to explain the most important concepts behind how it works. There is absolutely no claim of completeness; Kubernets is way too big for a blog tutorial like this.

Starting off

The easiest way to start using Kubernetes is Minikube. If you have an account with a cloud provider, and would like to first figure out the details of running a cluster on their platform, this tutorial will still work for you, as the commands work for any recent version of Kubernetes. See here for details on how to get Minikube running on your computer. In order to manipulate the Kubernetes mini-cluster minikube runs, you need the official CLI client named kubectl, which can be installed following the instructions on this page. You will also need Docker to create and push container images. Install Docker on your computer following the instructions here.

Once you have installed everything, make sure they are all available with the following commands:

kubectl version docker version minikube version

You can check whether Minikube is running using the following command, which also tells you whether there is an update available:

minikube status

If minikube is not already running, you can start it with minikube start . Normally, when you install minikube, it automatically configures kubectl to access it. You can check whether this is the case with kubectl cluster-info . Its output should be something like the following:

Kubernetes master is running at https://192.168.99.100:8443

If the IP is not in the 192.168.*.* range, or kubectl complains that configuration is invalid or the cluster cannot be contacted, you need to run minikube update-context to have minikube fix your configuration for you.

How is kubectl configured?

I think it is a good idea to shortly mention how kubectl is configured. Which API endpoints and clusters kubectl accesses are defined in the \~/.kube/config file by default. The file that is accessed can be changed with the KUBECONFIG environment variable, which should specify a list of paths, so if kubectl displays weird behavior whih you suspect might be due to the configuration, don't forget checking whether this environment variable is set. The kubectl configuration file is in the YAML format, like many other things in Kubernetes. It has two top-level keys that are of immediate relevance: contexts and clusters . The clusters list contains endpoint and certificate information for the different clusters to which the user has access. A context combines one such cluster with the user and namespace values for accessing it. One of these contexts is the currently active one; you can find out which by either looking at the config file, or running kubectl config current-context . You can also run kubectl config view command to show the complete configuration. You can limit the data shown to the current context with this command using the --minify option.

Nodes and namespaces

Two basic concepts that are relatively straightforward and can be explained without a lot of context are nodes and namespace. Nodes are the individual units of a Kubernetes cluster, be it a VM or an actual computer. What makes such a unit a node is the kubelet process that runs on it. This process is responsible for communicating with the Kubernetes master, and running the right containers in the right way. You can get a list of the nodes with kubectl get nodes . If you are using Minikube, and didn't do anything fancy with the configuration, there will be a single node. Nodes are not particularly interesting. You as a Kubernetes user will not be doing anything fancy with them, and cloud provisioners all have means of automatically or manually scaling the nodes in a Kubernetes cluster.

Namespaces provide a means to separate subclusters conceptually from each other. If you are running different application stacks on the same cluster, for example, you can organize the resources per app by putting them in the same namespace. A resource created without a namespace specified is created in the default namespace. It's not necessary to use namespaces, but they make certain things much easier, by helping you avoid name clashes, limit ressource allocation, or manage permissions. In case you start working with namespaces, and get annoyed by having to provide the --namespace switch to every command, here is a handy command that will set the default namespace for the current context:

kubectl config set-context $(kubectl config current-context) --namespace=my-namespace

Kubernetes dashboard

Kubernetes comes with a built-in dashboard in which you can click around and discover things. You can find out whether it is running by listing the system pods with the following command:

kubectl get pods -n kube-system

If there is an entry beginning with `kubernetes-dashboard`, it's running. In order to view the dashboard, first run the command kubectl proxy to proxy to the Kubernetes API. The Kubernetes API should now be available at http://localhost:8001, and the dashboard at this rather complicated URL. It used to be reachable at http://localhost:8001/ui, but this has been changed due to what I gather are security reasons.

Using a locally built image with Minikube

In the following tutorial, we will be deploying various container images in order to demonstrate Kubernetes features. Kubernetes uses Docker to retrieve and run container images, meaning that the usual rules of Docker container pull logic apply. That is, for a container image that is not available, if only a name and a tag are provided, Docker contacts the Docker Hub, otherwise hitting the registry in the container name. The aim of this tutorial is to get you to playing around with services running within a Kubernetes cluster as quickly as possible. Hence, the method I would recommend for accessing the container images from minikube is directing your Docker client to the daemon running inside minikube, instead of the local one. Configuring Docker to do so is straightforward with eval $(minikube docker-env) . Now, any image that you create and tag will be available inside minikube. You can make sure that this is the case by running docker ps . If the output contains a list of images from gcr.io/google_containers , you are doing it right. This proxy to the docker service in minikube will be valid only in the current shell; you will be back to using the local docker service when you switch to another shell.

If you are not interested in modifying and building the sample services yourself, you can also pull the sample images from my Docker.io profile. It should be enough to replace the kubetutorial prefix in the image tags with afroisalreadyin .

Running a service

Let's start off by running our first command to tell us whether there is anything running on the cluster. We will use the above mentioned kubectl client to do so, running the command kubectl get pods . What pods are will be explained in a second. As long as the client is configured correctly, as explained above, you should see only the message No resources found. What kubectl did was to access the Kubernetes cluster running within minikube as specified by the currently active context configuration and present the resulting information. kubectl is just one among many API clients; there are others, such as this Python client which is the other officially supported one. You can view the API requests kubectl is making by increasing the verbosity of the logging with the --v=7 argument, but careful, this will lead to a lot of textual output.

Kubernetes will not figure out for itself what we need to run, so let's go ahead and tell it to run a very simple application, namely the simple Python application from the Kubernetes demos repository. In order to do so, you need to first clone the repo, navigate to the subfolder simple-python-app , and create a container image by running the following command:

docker build -t kubetutorial/simple-python-app:v0.0.1 .

Once the build runs, you should be able to see it in the list of available images in the result of running docker images . After making sure this is the case, we are finally ready to run our first Kubernetes command, which is the following:

kubectl run simple-python-app \ --image=kubetutorial/simple-python-app:v0.0.1 \ --image-pull-policy=IfNotPresent \ --port=8080

It should be obvious that this command somehow runs the container that we just created, since the tag of the image is passed in with the --image argument. The imagePullPolicy=IfNotPresent argument tells Docker to use an existing local image instead of attempting to pull it. We are also specifying the port 8080 here as the port this deployment is exposing. This has to be the same port the application is binding to. Unless we provide this bit of information, Kubernetes has no way of knowing on which port to contact the application. Small side note: The demo service has to bind to this port on the general interface 0.0.0.0 and not on localhost or 127.0.0.1 .

How do we reach into Kubernetes to contact our service? This is the perfect time to introduce the most important abstraction in Kubernetes: The Pod. As with the other abstractions, pods are resources on the Kubernetes API, and we can list and query them using kubectl. Let's see which pods are now running, with the same command that we ran earlier, kubectl get pods . The output should closely resemble the following:

NAME READY STATUS RESTARTS AGE simple-python-app-68543294-vhj7g 1/1 Running 0 21s

Great, we have a pod running. But what is a pod, actually? A pod is the fundamental application unit in Kubernetes. It is a collection of containers that belong together, and whose lifetimes are managed together. These containers are deployed on the same node, their lifetimes are managed together, and they share operating system namespaces, volumes, and IP address. They can contact each other on localhost and use OS-level IPC mechanisms such as shared memory. The decision of what to include in a pod hinges on what serves as a single unit across the dimensions of deployment, horizontal scaling, and replication. For example, it would not make sense to put the data store and the application containers of a service into the same pod, because these scale and are replicated independently of each other. What does belong together with the application container is a container that hosts the log aggregation process, for example.

Now that we know what a pod is, and can figure out the name of our single pod running, we can query it using the kubectl proxy feature we already used above. Once the proxy is running, you can access the simple-python-app container on the port we specified in the previous command by querying the special URL that Kubernetes makes available for this purpose (don't forget changing the name of the pod at the end of the URL):

curl http://localhost:8001/api/v1/proxy/namespaces/default/pods/simple-python-app-68543294-vhj7g

We can also see the logs of our brand new pod with kubectl logs simple-python-app-68543294-vhj7g , which should show the stdout of our application. It is also possible to execute a command within the container, similar to the docker exec command, with kubectl exec -ti simple-python-app-68543294-vhj7g CMD . As with Docker, the -ti bit signals that a tty should be allocated, and the command should run interactively. The kubectl exec command allows you to pick which container to run the command in using the -c switch. When ommitted, the default is the only container in the pod, if there is just one, as per the definition of the pod.

Who created the pod?

It's nice that Kubernetes is running our container inside a pod, but we would still like to know where the pod actually comes from. We didn't tell Kubernetes to create any pods. In fact, pods are rarely created manually in Kubernetes. If that were the case, Kubernetes would not be offering anything new; the user would still be responsible for orchestrating the individual application units, and ensuring their availability. What the above kubernetes run command did was to create a Deployment. This can be seen by listing the deployments:

$ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE simple-python-app 1 1 1 1 1s

Deployments are one of the special kinds of resources in the Kubernetes world, in that they are responsible for managing the lifetime of application containers. These kinds of resources are called controllers, and they are central to the Kubernetes puzzle. You can get more detailed info about the new deployment with kubectl describe deployments simple-python-app . The describe subcommand is a very useful tool for getting detailed information on all resources. It also lists related resources, and events that concern the described resource. For this deployment, you can see a couple of things in the output of kubectl describe . First of all, there is talk of something called a pod template. This is what is used to create the pods when the deployment is being scaled, i.e. new pods are being created to meet the target.

What happens when we delete the pod? In order to view what is happening in real time, I would advise you to open a second terminal, and run the command kubectl get pods -w in it. The -w switch updates the output in regular intervals. Now, delete the existing pod with kubectl delete pod simple-python-app-68543294-vhj7g . In the output of the pod listing terminal, you should temporarily see a state like the following:

NAME READY STATUS RESTARTS AGE simple-python-app-5c9ccf7f5d-8lbb2 1/1 Running 0 4s simple-python-app-5c9ccf7f5d-kl77s 1/1 Terminating 0 43s

So as one pod is being deleted, another was already created (the status might also be ContainerCreating instead of Running . The responsibility for this recreation goes to Replica Sets. You can see the replica sets that belong to a deployment using the above mentioned kubectl describe command; the Replica Sets will be listed at the bottom, before the events. You can see that there are two lists: OldReplicaSets and NewReplicaSets . The difference between the two will be explained later in the context of rollouts. You can also list the replica sets with the kubectl get replicasets command.

Looking at the replica set created by our deployment with kubectl describe replicaset $REPLICA_SET_NAME , we can see at a glimpse a number of relevant rows:

# ... snip Replicas: 1 current / 1 desired Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: pod-template-hash=4035281104 run=simple-python-app-2 Containers: simple-python-app: Image: kubetutorial/simple-python-app:v0.0.1 Port: 8080/TCP Environment: <none> Mounts: <none> Volumes: <none> Events: <none>

This Replica Set is responsible for keeping one Pod with our simple-python-app container running, and it is doing that successfully, judging from the 1 current / 1 desired row. But as with pods, replica sets are intended to be created by Deployments, so you shouldn't have to create or manipulate them manually.

Short excursion on networking

As nice and useful as replica sets are, they not much of a help in terms of high availability. When a Pod goes down, another one is started, and it has a different name, a different IP address, and is possibly running on a completely different node. Also, what if we want to load balance these replicas? If Kubernetes were to offer service discovery only based on pod names, the clients of this service would need to do client-side load balancing, and keep an internal list of pods that need to be updated on every pod lifetime event. What about routing incoming traffic to services (ingress)? These are all pesky issues that need simplification. Kubernetes offers much easier mechanisms to achieve HA, load balancing and ingress. The basis for all this is the networking requirements Kubernetes imposes on the nodes and pods. These are the following:

All containers can communicate with all other containers without NAT (Network Address Translation).

All nodes can communicate with all containers (and vice-versa) without NAT.

The IP that a container sees itself as is the same IP that others see it as.

It is possible to use any one of various networking options that fit this model, with kubenet being the default. The above requirements sound relatively straightforward. One would think that each application container gets its IP. That is not the case, however, as it is not the application containers, but the Pods that get the IP addresses. Or in the words of the documentation:

Until now this document has talked about containers. In reality, Kubernetes applies IP addresses at the Pod scope - containers within a Pod share their network namespaces - including their IP address.

You can also verify that pods can be reached by IP on the exposed port by getting the private network IP address of the container with kubectl get pods -o wide . Afterwards, log on to the Minikube node with the command minikube ssh . From within this node, you can query the service with curl $IP_ADDRESS:8080 , which should return the response we have already seen.

How are pods that belong to the same replica set organized, in order to provide high availability, load balancing and discovery? The answer to this question is requires introducing another Kubernetes concept.

Services

I have been calling the tiny web application we have been using for demo purposes a service, but service has a totally different meaning in the Kubernetes world. A Kubernetes Service is an abstraction that allows loose coupling of pods to enable load balancing, discovery and routing. Through services, pods can be replaced and rotated without impacting the availability of an application. Let's start with a very simple example where we turn our simple Python application into a Service, which can be achieved with the following very simple command:

kubectl expose deploy simple-python-app --port 8080

If you now run kubectl get services , you should see a list consisting of two entries: kubernetes and simple-python-app . The kubernetes service is a part of the infrastructure, and you shouldn't meddle with it. The other service is what we are looking for, especially the IP address, which is listed under the column CLUSTER-IP . We are interested in this IP address because it is something special. It's a virtual IP Kubernetes has reserved for the new service. In the same output, you can also see that the port 8080 is exposed. We can now log on to the minikube VM (which is a Kubernetes node) with minikube ssh , and query what is now truly a service with curl $IP_ADDRESS:8080 , once more returning Hello from the simple python app . The network requirements mentioned above ensure the reachability of the service IP from the node.

Things get much more interesting when there are multiple pods in a replica set. In order to see the effect, let's use another service that provides more information in its response. This service is in the kubernetes-repository as env-printer-app . When the base path is called, it returns a print of the environment variables. Just like with the previous application, you can go ahead and create a container with the following command:

docker build -t kube-tutorial/env-printer-app:v0.0.1 .

We will start the Deployment with a replica count of 3, which will cause Kubernetes to start 3 pods right away. To do so, use the following command:

kubectl run env-printer-app \ --image=kube-tutorial/env-printer-app:v0.0.1 \ --image-pull-policy=Never \ --replicas=3 \ --port=8080

Now let's create a Service by exposing this Deployment with the following command, which is a slight modification of the expose command we used earlier:

kubectl expose deploy env-printer-app --port 8080

A new service env-printer-app should pop up in the output of kubectl get services . Note the IP address for this service under CLUSTER-IP as $IP_ADDRESS , and log on to minikube via ssh again. Afterwards, run the following command a couple of times:

curl -s $IP_ADDRESS:8080 | grep HOSTNAME

This command makes a request to the service endpoint, and filters the HOSTNAME environment variable out of it. You should observe that the hostname alternates between the various pod names. Kubernetes is distributing the requests among the replica pods for us, giving us load balancing out of the box.

This very short demo of services leads to more questions than answers. How does the service know which pods to hit when a request comes in, for example? Why can we contact our service only from within the cluster? How can we enable external access to it? Before we can answer these questions, however, we need to have a look at a better way of specifying deployments, services and other resources.

Using the command line versus manifest files

Until now, we have been using the command line interface to Kubernetes via kubectl . It is possible to get quite far with kubectl , as it is pretty complete, but it can become difficult to read, share with others, and organize in a repository. A much better method for organizing Kubernetes resources which adheres to the infrastructure as code mantra is using manifest files. These are either YAML or JSON files (although YAML is preferred) that specify in a more structured format the resources to be created and actions to be undertaken. A manifest file takes the form of a list of resources of different kinds, together with metadata and a spec. It is also common and recommended practice to specify the version of the API that is targeted with each entry. The different entries must be separated with a triple dash separator, which signifies the start of a new document in YAML. This separator is mandatory; if you leave it out, only the first item in a list will be processed.

The resource specifications are documented in great detail in the Kubernetes API documentation. What's even better, however, is that the kubectl command is self-documenting. To get documentation on pods, you can use the kubectl explain pods command. This command will print, prefixed by a short description, the various fields a pod manifest can contain. In order to go deeper in this tree, you can run commands such as kubectl explain pod.metadata.labels , which will give more detailed information on individual fields.

If you have a look at the entry for deployment in either the online or command line documentation, you will see that the metadata field is same across all resources, and the name field is required. This field enables us to refer to resources in commands when we want to get detailed information or delete them, or cross-reference from other manifest files. The spec field is required to adhere to the DeploymentSpec configuration, which should have a template field that describes the pod to be deployed. This template, in turn, must have a metadata field itself, and a spec that should contain a list of containers. As per this specification, here is how to create the above deployment example for the env-printer-app , in YAML format:

apiVersion: extensions/v1beta1 kind: Deployment metadata: name: env-printer-app spec: replicas: 3 template: metadata: labels: app: env-printer-app spec: containers: - image: twyla.io/env-printer-app:v.0.0.1 imagePullPolicy: IfNotPresent name: env-printer-app

It is possible to see a common pattern of nested resources that all have metadata which is used to refer to each other, templates that tell Kubernetes what kind of resources to create, and various other kinds of auxiliary information, such as the replicas field. You can now go ahead and use this YAML file, saved into deploy.yaml in the kubernetes-repository/env-printer-app directory, to create a deployment by running kubectl apply -f deploy.yaml . It is possible to create all resources in a directory by kubectl apply -f with the directory path.

You can also use kubectl get KIND NAME -o yaml to get a detailed description of a resource in YAML format. This YAML document might include much more than the information you supplied when creating a resource, as the values for the defaults you omitted, and those calculated or set by Kubernetes are also included. Another really great feature that relies on the YAML representation capabilities of Kubernetes (one of my favorite features) is editing a resource with the command kubectl edit KIND NAME . This command will fetch the resource description in YAML, and load it in the editor defined by the EDITOR (or KUBE_EDITOR , if it's defined) environment variable. Once you save your changes and exit, the new resource description will be applied to the resource. This is a great way to try things out quickly without having to keep multiple versions of resource definitions.

Services, continued

Alright, where were we? So we have a bunch of containers running in Pods, provisioned and kept alive through Deployments, bundled into a Service that puts them behind a common IP. And we can put all of these into one or more YAML files to recreate them arbitrarily. This is a good point to explain one very interesting and versatile feature of Kubernetes: Selectors. If you go ahead and get the details of the env-printer-app service we have created above with kubectl describe service env-printer-app , you should see a row that begins with ~Selector: ~. This selector configuration tells you how Kubernetes finds the pods it should collect behind the virtual IP of the service. If you didn't do anything funky in the meanwhile, the value of the selector row should be run=env-printer-app . If you describe the deployment targeted by this service with kubectl describe deploy env-printer-app , you will see exactly the same selector line. Services and deployments use the same mechanism to match the pods that they hit or control. Which pods are these? This question can be answered by filtering a search by label, as in the following command:

kc get pods -l run=env-printer-app

Not surprisingly, these are the three pods created by the original deployment. This selector-based mechanism is used by many components in Kubernetes, and it is very versatile in that it allows custom labels. This opens up a whole lot of possibilities for different patterns, such as A/B deployments, rolling updates (which we will see later) and similar things.

What is thus happening is that a collection of pods, as picked by the spec.selector attribute, is exposed as a service on an IP. This is not the only way to expose a service, however: There are different kinds of Services based on how this exposing happens. The default is the ClusterIP kind, which is what we have now. Other kinds are NodePort , where a service is exposed on the same port on all exposed nodes, LoadBalancer that uses a platform-native load balancer to expose a service to the outer world, and ExternalName which enables you to provide an external service on the local cluster as if it's an internal one.

These all have their use cases, but the ClusterIP service is the one that covers the most use cases, so we will concentrate on it here. Having multiple pods behind a single IP solves many problems, since Kubernetes also takes care of things like load balancing (done by randomly routing requests; a new proxy mode will introduce more options) or managing modifications in target pod set. One thing it does not solve, however, is the problem of figuring out this IP in the first place. This is another point in which Kubernetes shines: Matching a name to an IP address is done using DNS on the internet, and Kubernetes builds on this common protocol by providing an internal DNS service itself. By default, a DNS A record is created, pointing to the service IP, for each ClusterIP service. Hence, we should be able to refer our env-printer-app under this exact name. To see that this is the case, run the following command to run bash on a container:

kubectl run my-shell --rm -ti --image cfmanteiga/alpine-bash-curl-jq bash

There are quite some arguments to this command, which need some explanining. The --rm switch tells kubectl to delete the deployment and the pod once the command is run, while -ti asks it to attach a tty to the container, and make it connect to the stdin of the container process. The --image argument specifies a lightweight alpine-based image with some debugging utilities, and the last argument is the command to use instead of the entry point of the container. In the shell that starts, you can now run curl http://env-printer-app , and enjoy the environment varliable list delivered by the service.

Ingress

Our service is now humming in the cluster, accepting requests when we hit it at http://env-printer-app. In order to make it available to the outer world, we need to do one last thing: Tell Kubernetes to route HTTP requests from the outside to a certain location to this service. This process is called Ingress, and Kubernetes offers a complete system to handle it. There are two things you need to enable to route requests to the env-printer-app from the outside:

An Ingress controller, essentially a reverse proxy running within Kubernetes that can be configured using Kubernetes-native resources. The two built-in solutions are GCE and Nginx-based. In order to use the Nginx-based ingress controller on Minikube, you have to enable the extension with minikube addons enable ingress .

Ingress specifications. These are resources just like Pods and Deployments, and contain information on how to map incoming requests to services, serving as configuration for the aforementioned ingress controller.

An Ingress specification for the env-printer-app is included in the sample project repo as ingress.yml . After activating the minikube ingress plugin, you can run kubectl apply -f ingress.yml to create an ingress that maps requests to http://env-printer to the env-printer-app service. In order to test the ingress, you need to first figure out the IP of the minikube VM with minikube ip , and then edit /etc/hosts on your computer, adding the line $IP_ADDRESS env-printer . You should now be able to navigate to http://env-printer in your browser, and see the output of the env-printer-app service.

Once you have a deployment managing a set of pods, there are a couple of things you can do with it to adapt to new conditions. First of these is scaling the set of containers to meet load conditions. One way of achieving this is using the kubectl scale command, as follows:

kubectl scale deploy env-printer-app --replicas=4

Alternatively, you can use the kubectl edit deploy env-printer-app command to bring up an editor, and change the spec.replicas field to the required number. If you now run kubectl describe deploy env-printer-app , there should be a new scaling event in the Events section. When the number of replicas is changed, Kubernetes simply creates new pods, or terminates existing ones, without any further complications. It's a different situation when the container spec for a deployment is changed, however. Kubernetes, based on the strategy specified by the user, replaces the pods progressively, to enable a smooth transition from one set of pods to the other. This is called rolling updates.

In order to demo rolling updates, I added another project to the sample Kubernetes services repository, the rollout-app . You can go ahead and create the service by running kubectl apply -f deploy.yml --record in the app's directory, which will create the deployment, the service, and the ingress. The reason for the --record switch will be explained in a couple of paragraphs. If you edit your /etc/hosts file to add http://rollout-app with the minikube IP, you should be able to navigate to this URL and see a big display of the port's hostname.

If you open rollout-app/application.py , you can see two peculiar things there. One is the /healthz endpoint that returns a simple OK message and nothing else, and the other is a time.sleep(5) before the app starts. The purpose of the /healthz endpoint might become clearer if you also look at the deploy.yml in the same directory; this endpoint is registered as a readinessProbe on the deployment. The readiness probe is a part of the pod lifecycle system of Kubernetes. Before this probe is valid (for HTTP probes, it must return a status code between 200 and 400), the new pod is not marked as "ready", and requests will not be routed to it. Due to the sleep of 5 seconds before our application is started, the pods of the rollout-app will not be ready for at least five seconds. Now let's have a look at how this delay interacts with the rolling updates feature of Kubernetes. Once you have deployed the application, change application.py in some minor way, such as adding a newline. Afterwards, create a new docker container with a new tag with docker build -t kubetutorial/rollout-app:v0.0.2 . . Then go ahead and change the Docker image for the rollout-app deployment to the new version with the following command (again with the --record switch which will be explained later):

kubectl set image deploy rollout-app rollout-app=kubetutorial/rollout-app:v0.0.2 --record

Kubernetes gets to work right away, creating new pods and terminating the ones these are supposed to replace. You can see that this is the case by running kubectl get pods . One peculiar (or actually nice) thing is that Kubernetes does not just pull down the running pods, starting their replacements at the same time. A rollout process is applied, whereby new pods are created as old ones are taken down. You can follow this process by running the command kubectl rollout status deploy rollout-app . This command will hang with a message like Waiting for rollout to finish: 2 of 3 updated replicas are available…. So now the deployment is in the middle of a rollout process. We will see where these numbers come from later. A rollout is actually the process of moving from one replica set to another. You can see that this is the case by running the command kubectl get replicaset (or replace replicaset with rs to make the command shorter). You should see two replica sets that begin with replica-set , one belonging to the old state, and the other belonging to the new state. The DESIRED, CURRENT and READY values of one should decrease, while the other one goes up and approaches required values.

One thing you can do is pause this rollout while it is in progress with kubectl rollout pause deploy rollout-app . This will leave the pod counts the way they are when you run the command, and give you the chance to run checks, to make sure everything is OK. Let's say that you start a rollout, pause it to run some checks, and discover that you made a mistake, and would like to rever to the previous version to fix the issue. This can be achieved by rolling back the rollout with kubectl rollout undo deploy rollout-app . But let's say that you want to move back even further in the deployment history. This is where the --record switch to the kubectl apply command comes into play. Thanks to this switch, we can now see the commands that caused a rollout on this deployment, and a version number that we can use to refer to that rollout. After you deploy version 0.0.2 of rollout-app , the output of the kubectl rollout history deploy rollout-app should be similar to the following:

REVISION CHANGE-CAUSE 1 kubectl apply --filename=deploy.yml --record=true 2 kubectl set image deploy rollout-app rollout-app=kubetutorial/rollout-app:v0.0.2 --record=true

You can switch e.g. to revision 1 with the following command:

kubectl rollout undo deploy rollout-app --to-revision=1

The rollout feature of Kubernetes is very well-designed and feature rich. Other things you can do are precisely control the number of percentage of pods that are replaced, or set conditions on failing rollouts so that they can be rolled back automatically by other tools.

Going further

Until now, I have been singing Kubernetes' praise, but not everything about it is perfect, unfortunately. We have run into a couple of issues building a Kubernetes cluster. Kubernetes, despite being a relatively young project, is under heavy development, and keeping up with it is not a simple job. The development process is very well-managed, but nevertheless it is a full-time responsibility to keep up with the changes. This situation is mirrored on the provider side of things, as cloud vendors are racing to provide the best hosted Kubernetes solution possible, which also leads to considerable trial-and-error. Azure, for example, started off with a feature called ACS, which was supposed to be a generic container management solution, but quickly recognized how popular Kubernetes was coming, and deprecated ACS in favor of AKS which is directed solely towards Kubernetes, and has extra features such as redundant master nodes. Unfortunately, we are on ACS, and need to make the move to AKS at some point.

Another thing you have to keep in mind when running Kubernetes is that it has significant platform-dependent parts, and these are not uniform in terms of correctness and reliability. A short time after moving to Kubernetes on Azure, we found out that there was a serious bug with Kubernetes on ACS that makes the storage mounting feature of Kubernetes nearly unusable. Our solution is to rely as much as possible on the cloud offerings of Azure such as CosmosDB and managed PostgreSQL, but we will need to use local storage in a service at some point. Fortunately, the bug appears to be fixed in Kubernetes 1.10.

As Kubernetes increases in feature set and complexity, tools built on Kubernetes to simplify workloads and provide more integrated workflows have also started popping up. Kubernetes was never meant as the last application level, meaning that there will be tools that build up on it for specific developer workflows, which is already happening. It looks like Helm is the most popular choice on this front, but there are other alternatives such as OpenShift. So be prepared to learn another tool that runs on top of Kubernetes in the near future.

Bonus: Shell Helpers

There are a couple motions you repeat over and over when you are working on a Kubernetes cluster. One of these is getting the name of a pod. As the pod name is derived from the name of the deployment, you end up running kubectl get pods and either grepping it searching it visually. In the case of single-pod deployments, fetching the name of the pod is very eash with the following bash function:

function podname { kc get pods | grep $1 | awk '{print $1}'; }

If you want the name of the simple-python-app pod, for example, you would need to run something as simple as podname simple . You can also use this function as argument to other kubectl commands, e.g. to print the logs with kubectl logs `podname simple` .

Another handy snippet (written by my Bash Jedi Master friend Matthias Krull) is the following, which lets you switch between Kubernetes configurations like between Python virtual environments:

function kubeon { if [ "${1}" ]; then local config_file="${1}" else echo "Usage: kubeon <config|config_file>" return 1 fi if [ ! -f "${1}" ]; then config_file="${HOME}/.kube/${1}" fi if [ ! -f "${config_file}" ]; then echo "No config file found. Tried ${1} and ${config_file}" return 1 fi export KUBECONFIG="${HOME}/.kube/${1}" export KUBEON_PROMPT="${1}" export KUBE_MASTER=$(kubectl config view|grep server:|cut -d/ -f3) }