Photo by chuttersnap on Unsplash

Recently I had the privilege of presenting my latest findings of Kubernetes Federation v2 at the OpenInfra Days in London. While investigating the subject, I realised there are only a handful of resources available to help you get started. Henceforth, this blog post aims to provide a guide on incipient stages of experimenting with Federation v2.

Before I deep dive into the topic, I would like to present the use case for federated clusters at Condé Nast International. For over a century Condé Nast International has set the benchmark for print and digital publishing. With brands like Vogue, GQ, Wired, Condé Nast Traveller under our umbrella we are operating in more than 16 markets across the globe. Hence, having a scalable, fault tolerant and highly available platform combined with a robust delivery process is the raison d’etre of the cloud platform team. Currently, we are hosting multiple Kubernetes clusters and application management can easily become a tedious process. To enable the creation of new clusters with minimal effort, we are investigating Federation v2 and the features it provides.

The Theoretical

If you have a considerable customer base with an infrastructure formed of multiple clusters (similar to our use-case at Condé Nast International) and starting to consider a centralised mechanism to deploy consistently your application, then Federation v2 is probably what you want to look into.

Federation v2 aims to provide a control plane capable of interacting with multiple clusters, and manage the propagation of services and applications based on the user defined policies. Federation v2 makes a strong use of CRDs (Custom Resource Definition) that reuses available master API endpoints and results with building blocks that can be extended to any Kubernetes resource.

In simple terms, Federation v2 is a powerful mechanism that can deploy and manage applications and services in a multi-cluster environment.

To have a working federation control plane, it is required to have the cluster registry installed. Cluster registry is a tool that stores a list of clusters and the configuration required to access those clusters. Once a cluster is known to the registry, it is possible to add that cluster to the federation.

With Federation v2 higher level behaviour management is introduced as a concept. This allows to comply with requests like “we want a total of 1000 replicas of this application running at all times across all available clusters”. The placing engine in tight coupling with the user defined policies will be able to make sophisticated decisions to address where a resources should be scheduled. Additionally, this unlocks the possibility of migrating federated Kubernetes resources across clusters for incidents where one or more clusters are lost.

Additionally the higher level APIs provides support for multi-cluster service configuration. It programmatically manages DNS records and introduces a process for pods and external clients to discover a federated service.

For more details about Federation v2 architecture and latest features visit sig-multicluster.

The Practical

Federation v2 requires clusters hosting Kubernetes v1.11 or a higher version. In this example, we will use kind, which simplifies the provision of Kubernetes clusters on a local machine. To spin up a kind cluster use the following command:

$ kind create cluster --name “test-cluster" --image kindest/node:v1.12.5

Note: Federation v2 also provides a script to spin multiple kind clusters.

Once 2 or more clusters are available, chose one cluster to host the controller manager for federation and the cluster registry. A federation-v2 helm chart is available to install the all the necessary components:

# cluster registry will reside within kube-multicluster-public namespace

$ kubectl create ns kube-multicluster-public # install federation v2

$ helm install charts/federation-v2 --name federation-v2 --namespace federation-system --set clusterregistry.enabled=true

If you have a brand new host cluster, tiller needs to be installed before invoking any helm commands. To do so, use the following commands:

$ helm init # service account for tiller

$ kubectl create serviceaccount --namespace kube-system tiller

$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

$ kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

At this point we have all the pre-requisites checked to join available clusters using kubefed2 command line tool. In this example kubefed2 v0.07 is utilised:

# join cluster1 to the federation

$ kubefed2 join cluster1 --cluster-context cluster1 --host-cluster-context cluster1 --add-to-registry --v=2 # join cluster2 to the federation

# note: cluster1 is the hosting cluster of the federation controller $ kubefed2 join cluster2 --cluster-context cluster2 --host-cluster-context cluster1 --add-to-registry --v=2

And that’s it! The end result is 2 federated clusters, with the federation control plane running in host cluster. The default endpoints for federated resources (e.g. federateddeployment, federatedservices, federatednamespace etc.) are enabled and ready to be used. To start deploying into the federated clusters, follow the this example.

Conclusion

Federation v2 is an incredibly powerful mechanism that facilitates the process of managing application and services on multiple clusters. The building blocks approach allows the extension of federation to supported and custom resources, which only enhances the flexibility Federation v2 is providing for future development.