Turns out you can use good old BGP to power your Kubernetes network! In this post I’ll cover how to build your Kubernetes network with BGP and how to use it for traffic engineering in your Kubernetes clusters! My hope is that this post gives you enough context to use BGP in your own Kubernetes clusters (where it makes sense), especially for those who are running Kubernetes on-premise.

Why BGP?

BGP (Border Gateway Protocol) is an industry standard for exchanging routing and reachability information among systems. BGP is an essential piece of the public internet as large networks across the globe are glued together using BGP. If you want to dig deeper into the specifics of BGP, I recommend checking out the IETF RFC on it.

If you are running Kubernetes on the cloud (AWS, Azure, DigitalOcean, GCP, etc), then using BGP likely won’t make sense. Your cloud provider either provides a SDN for your cluster or in some cases you may be stuck using an overlay network on top of the network that your VMs are running on. However, if you are running Kubernetes on-premise and you have the flexibility to configure your network, then BGP is one of the many options you can use to setup routes for your cluster. Using BGP opens up a lot of possibilities for your Kubernetes network such as routable Pod and Service IPs which you wouldn't be able to get in most cloud environments (yet).

How to Setup your Cluster with BGP

Configuring your Networking Devices

This is the shameful hand wavy part of my post where I say “it depends”. And in reality, it really does. In some cases, there may be a lot of configuration required since there are so many knobs we can tune with BGP. On top of that, there are thousands of different network devices from various vendors that will change the steps required to configure your network correctly. For this step, I highly recommend consulting a network engineer. The remainder of this blog will assume that you know how to configure routes, import/export policies, etc on your network devices.

Configuring your Kubernetes Cluster

What’s nice about using BGP on Kubernetes is that building your own solution is doable. There are rich APIs in various languages for using both Kubernetes (client-go, client-python, etc) and BGP (gobgp, exabgp, etc). If you don’t want to build your own though (understandable), there are a few projects out there already. My favourite among them is kube-router due to its ease of use, but other solutions like MetalLB and Calico may be a better fit for your use case.

kube-router offers three main features: service proxy with IPVS, network policies, and network routing. For this post, we’ll assume we only want to deploy kube-router with networking routing. To do so, deploy kube-router like so: