One of the great challenges associated with cloud computing at scale is how to manage the connection load when millions (and even billions) of devices are maintaining always-on cloud connections. We developed software to address this issue in our cloud-enabled X1 video platform, and today we’re excited to announce that we’ve made a version of the software available open-source.

We built Redirector because we weren’t finding available tools that supported the effective, scalable management of the sort of persistent, long-standing cloud connections that our cloud-native X1 platform requires.

Redirector is a server traffic-shaping tool designed to address two challenges with traditional load balancers: First, long-term connections are often maintained in hardware load balancers that were not able to handle scale in the order of millions of simultaneous connections. Second, even though such load balancers are able to distribute connections evenly amongst large groups of servers, they lack they ability to direct internal test clients to specific versions of servers and to release new software to backend servers in a controlled manner.

In building the Redirector server traffic-shaping tools we focused on several core areas:

Scalability : Ability to support millions of client connections connecting to thousands of backend servers

: Ability to support millions of client connections connecting to thousands of backend servers Low Latency : Ability of client connections to quickly establish connections with backend servers

: Ability of client connections to quickly establish connections with backend servers Service Discovery : Ability to easily register thousands of backend servers with a service discovery mechanism along with self-identifiers such as Service Name and software version

: Ability to easily register thousands of backend servers with a service discovery mechanism along with self-identifiers such as Service Name and software version Rule Based Traffic Shaping : Ability to leverage specific versions of backend applications and release backend software versions in a controlled manner

: Ability to leverage specific versions of backend applications and release backend software versions in a controlled manner Fault tolerance / high availability : Resilient to software and hardware failures

: Resilient to software and hardware failures Custom protocol support : Ability to support different communication protocols such as HTTP / HTTPS / WS / WSS

: Ability to support different communication protocols such as HTTP / HTTPS / WS / WSS Pluggable security framework for configuration management : Ability to enforce authentication and authorization for the rules / configuration management

Design

Redirector Framework

The Redirector framework is a dynamic context-based gateway tool. The Redirector uses service-discovery in combination with a rules engine to define the particular version of a given application that the client needs to be redirected to and returns an IP address of particular application.

Redirector Gateway

The core component of Redirector system is Redirector Gateway. It receives requests from clients and returns information about a particular host that the client should connect to. Redirector Gateway uses a distributed key/value store (Zookeeper) as source of truth for service discovery and Redirector WS for rule engine facilities. In situations when the source of truth is down, Redirector Gateway uses its local file system copy of Last Known Good service discovery and rule engine data.

Redirector Model

The Redirector Gateway processes rules based on the following model:

Default Server (specifies default version of application which we redirect to)

Traffic Distribution Rules (specifies how many %% of traffic will be redirected to particular version of application)

Flavor (Version) Rules. Match particular context (http headers or other information coming from client device) to version of application or stack with version deployed or to final URL.

URL Rules. Flavor Rules return template of URL and then URL Rules define concrete values for template: protocol, port, URL path, IP protocol version (4/6)

Whitelist. Gateway discovers application instances only from stacks included into Whitelist

Namespace Lists - value lists, e.g. list of mac addresses or IP addresses or service account ids which are used by Flavor and URL rules

The Redirector Gateway considers all these constituents as a single aggregate called Redirector Model. Redirector Gateway uses separate model for each particular application. The Redirector Gateway updates the models on notification sent from Redirector Web Service through Zookeeper. When the Redirector Gateway receives a notification, it pulls the Model and uses it until the next notification. Every time the model is updated, it is also persisted in the local backup. Even if the Redirector Gateway is restarted, the persistent Last Known Good model facilitates the functioning of the Redirector Gateway with reasonable expectations even when the sources of truth may not be available to load the most recent models.

Redirector Service Discovery

The Redirector Gateway is designed to return IP address for any particular version of the application based on the rules using service discovery pattern. There are 2 modes currently supported for service discovery. One is static mode, and the other is dynamic mode.

In Static service discovery mode, during model update the Redirector Gateway takes a snapshot of all service discovery data from zookeeper, keeps it in-memory and then uses it until next model update. In Dynamic service discovery mode, as long as the host is registered in zookeeper it is available to be returned back by the Redirector Gateway. If the host is not registered in zookeeper, it will not be returned back as a potential target host.

Redirector Web Service

The Rules are managed by Redirector Web Service (Redirector WS) through an Admin UX. Redirector WS saves rules in Zookeeper and then Redirector Gateway loads new rules from there.

We strongly encourage everyone to use, improve and contribute back to the effort. You can download the source code at https://comcast.github.io/redirector/

Redirector could not have been possible without the work of many engineers and architects, drawing on feedback from our internal users across the organization.

(As a note, this blog describes the Redirector open-source software and its capabilities and does not describe specific services or approaches deployed by Comcast.)