Evaluating Ingress Controllers

Ingresses are critical to any successful Kubernetes (k8s) deployment. Ingresses allow you to define how external (and/or internal) traffic is routed to services within your cluster. The Kubernetes documentation states that:

“An Ingress can be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting.”

However, Ingresses themselves don’t do anything — they’re just metadata. The heavy lifting is performed by Ingress Controllers. An Ingress without an Ingress Controller won’t do anything. There’s one more catch: while there are a number of system controllers (like ReplicaSet Controller, Endpoints Controller, Namespace Controller and others) that are managed by the Kubernetes control plane, Ingress Controllers are not automatically started with a cluster — you have to install, configure and manage your own Ingress Controllers.

It’s also possible to have multiple Ingress Controllers in the same cluster. You can divide the “routing space” by using Ingress class annotations, so that each Ingress knows which Ingress Controller should be handling it. You could end up using a combination of Ingress Controllers for different scenarios within the same cluster. For example, you may have one Ingress Controller for handling external traffic coming in to the cluster including bindings to SSL certificates, and have another internal Ingress Controller with no SSL binding that handles in-cluster traffic.

There are scores of Ingress Controllers to choose from. The Kubernetes documentation has a list of popular Ingress Controllers here.

These controllers have varied feature sets and varying levels of community or commercial support. Some are “pure” edge routers, while others have features more akin to service meshes.

Selecting the Right Ingress Controller

So how do you choose the right Ingress Controller? There are several criteria that are important to consider when doing so. In this post we’re not going to do a feature comparison between specific Ingress Controllers, since there are many head-to-head comparisons on the internet already, and because there are so many Ingress Controllers. Rather, we’re going to discuss what features you should be weighing when selecting an Ingress Controller.

1/ Traffic Protocol

Are you just routing HTTP(S), HTTP/2 or websockets? Do you want to route TCP/UDP or gRPC? Not all Ingress Controllers support all these protocols, so you’ll have to check which protocols an Ingress Controller supports.

Do you require zero-downtime configuration changes — often called “hitless reloads”? Some Ingress Controllers require downtime in order to update configuration, while others update dynamically without downtime.

3/ Resiliency

Do you need rate limiting, retries or circuit breakers at the edge, or have you built this functionality into your services yourself? Some Ingress Controllers support these features, which means you won’t have to code them yourself.

4/ External Load Balancer Integration

Are you integrating with an external, managed cloud-based load balancer? Make sure the Ingress Controller you select integrates well with your external load balancer in order to reduce work and management for your networking team.

5/ Service Mesh

Ingress Controllers can be configured to handle external traffic (traffic originating outside the cluster) or internal traffic or both. If you need to observe or trace internal traffic, you may need a special breed of Ingress Controller — a Service Mesh. Kubernetes provides standards for Service Meshes through the SMI Specification for interoperability. If you do require a Service Mesh, make sure you pick the right tool for the right job. Ingress Controllers and Service Meshes are not mutually exclusive.

6/ API Gateway

Do you need an Ingress Controller or an API gateway, or something that does both? Typically API gateways integrate business logic, while edge routers are typically business agnostic. For example, API Gateways let you monitor traffic per customer, or measure transactions for billing purposes. If you require business logic at the edge, you should probably look at an API gateway instead of an Ingress Controller. Just as with Service Meshes, Ingress Controller and API gateways are not mutually exclusive.

7/ High Availability

Can you afford downtime when a server restarts for either planned or unplanned maintenance? If so, you need high availability for your Ingress Controller. Not all Ingress Controllers support high availability.

8/ Load Balancing Algorithms

What sort of algorithm-based routing do you need? Most Ingress Controllers support Round Robin, but if you want Least Connection so that the load on your services is taken into account, you’ll need an Ingress Controller that supports more advanced load balancing algorithms.

9/ Advanced Traffic Shifting

Do you need to perform canary testing (shifting a percentage of traffic to different services for progressive exposure)? Load balancing lets you spread the load of a service, but not all load balancers can split traffic using more sophisticated rules. If you’re testing in production using techniques like canary testing, make sure the Ingress Controller you select supports traffic shifting.

10/ Resources Constraints

Are you cost sensitive as far as resources in your cluster are concerned? Ingress Controllers can be resource intensive, so if you are cost sensitive, you’re going to be better off with a light-weight Ingress Controller. Some Ingress Controllers support scaling up and down, while others do not.

11/ Monitoring

Do you need to integrate with existing metrics and log collection systems? Some Ingress Controllers offer limited monitoring and logging and may not support your specific monitoring and logging tooling.

12/ Support

Do you need enterprise support? Open source Ingress Controllers are easy on the check book, but what happens when you need support in the middle of the night? Remember, some open source Ingress Controllers offer enterprise support plans.

13/ Ecosystem

Make sure the Ingress Controller you are considering has support in the Kubernetes partner ecosystem.

Conclusion

As you can see, there are many factors that require thought before you can select the right Ingress Controller for your clusters. Don’t just select a hype-driven, popular option — be deliberate about your requirements and then evaluate Ingress Controllers according to the criteria we’ve listed. If you do, you’ll have made an informed decision about a very important part of your infrastructure!