One of the most important pieces of any modern web application is the network. As applications become more distributed, it becomes crucial to reason about the network and its behavior in order to understand how a system will behave. Service meshes are more and more frequently proposed as a means of tackling this problem. If you’re not familiar with meshes, Matt Klein has a great intro to them, and Christian Posta has a great series on Patterns with Envoy.

Fundamentally, modern apps benefit from networking patterns like meshes for three reasons:

Scale: At the scale of most modern web applications, your traffic is a thing you manage. You can split off 1% of it for a release. You pay for it between data centers. 100% of it could swamp a new service. Microservices: Whether you have tens or thousands of services, microservice communication goes over the network. This makes the network a good point of observation, and pairing control with observation is quite powerful for any operator. Elasticity: Dynamic infrastructure means configuration is better implemented as a set of dynamic rules than as a static description. It’s not important that a server has a particular home on the network, only that the necessary traffic can be routed to it while it’s alive.

Harnessing Your Traffic

The best approach to managing all this traffic is to run it through a high-availability cluster of smart proxies. But what does a deployment actually look like?

One common pattern is to deploy sidecars next to every service instance (server, container, lambda function, etc.). All your code then talks to localhost, both inbound and outbound, and the proxy sends it to the correct destination proxy or external service. If you’re using an orchestrator like Kubernetes, you can inject sidecars into a pod spec pretty easily.

Alternatively, you can use a proxy pool, where you spin up a number of instances of the same proxy, all configured in the same way. You may only need one instance, but it’s prudent to configure two or more in a high-availability pool to avoid a single point-of-failure. As with the sidecar model, all inbound traffic is pointed to this pool, and the proxies’ dynamic configuration routes traffic appropriately.

There is one big missing piece to using sidecars, though: sidecars don’t route traffic from your customers to your services.

The Case for Proxy Pools

No matter what, you’re going to need a proxy pool! If you’re running in the cloud, your infrastructure provider probably provides a simple-but-fast network load balancer at the edge of your network. Amazon’s ELB, Google CLB, and Microsoft’s ALB all offer a common set of features, including DDoS protection, SSL certificate management, and the ability to expose only a single IP for your whole site. Beyond that, you’ll need some facility for traffic routing to direct different URLs to different services. Though Kubernetes didn’t invent the word, it has popularized the term ingress for any sort of service that maps traffic from outside the cluster to internal services. It’s an important concept for any app, and a proxy pool gives you a single place to manage ingress to your services.

Secondarily, deploying a proxy pool is simpler and therefore less dangerous. Since it’s a separate service, you can start running it without affecting any of your existing traffic. You can even test it out by typing in the IP address manually, before any customers see it! Unlike a sidecar, proxy pools can start sending upstream traffic to individual services with no change to the service. Updating the downstream behavior is a matter of configuring them to point to the proxy pool, and it can be done incrementally and safely. If you discover that the proxies in your pool need a different bootstrapped config, you only need to update the proxies in the pool, instead of every single sidecar across the whole cluster.

When Should You Use a Sidecar Too?

If your network is complex enough that you have to manage it directly, moving to sidecars for internal traffic will probably be a net win.

For example, an app that’s deployed across multiple datacenters will need a pool in each region. It’s easy to make the case that a single pool is straightforward to reason about. It’s harder to claim that five high-availability pools in one cloud provider plus another dedicated seven for your HIPAA-compliant team is similarly simple.

Specifically, there are several benefits to using sidecars:

One fewer network hop for internal calls

The ability to impose guaranteed policy on intra-service communication

A place to inject out-of-process authentication and encryption

Guarantees for global state, like rate limiting

Ability to isolate policy and configuration changes to subsets of services

When it comes to the larger challenges described at the beginning of this post, these are secondary benefits. You can get most of the advantages of a modern traffic management stack by deploying a proxy pool first. As you convert all your services over to using that pool and gain operational muscle memory, you can easily upgrade your intra-service communication to a mesh using sidecars. In the meantime, though, don’t be scared of just deploying a set of proxies at the edge and running some of your services through that pool. Incremental improvement is powerful, and pools give you that flexibility.

If you want to use traffic shifting and network-derived metrics to make releases less stressful, sign up for your 30-day trial of Houston, by Turbine Labs. It supports both proxy pool and sidecar configurations, with powerful configuration and observability in all your favorite environments. Get started today!