Customizing Kubernetes DNS using Consul to scale Nginx RTMP

Making Nanit’s camera streams first-class citizens of Kubernetes

At Nanit, we’re running a very large cluster of Kubernetes with lots of nginx-rtmp pods to support our customers’ live feeds of their baby’s cribs.

Running lots of nginx-rtmp pods imposes a scaling problem:

A user who’s trying to watch their baby’s Nanit camera might end up connecting to an nginx-rtmp instance while the camera is streaming to another nginx-rtmp instance. This can occur because we have our nginx-rtmp pods behind a Kubernetes service which balances the load.

User connected to a different Nginx-RTMP than the camera

To solve this, we can pull the stream from the Nginx publishing the camera’s stream to the Nginx connected to the user using nginx-rtmp’s pull method. But how do we know which Nginx instance has our stream?

This is where kube-dns and Consul come into play. We will use DNS for stream discovery. Every stream will have a DNS record and we will query our DNS service for the stream we are looking for. We will use Consul to handle our stream DNS records and will customize kube-dns to route DNS queries to Consul.

Pulling (relaying) from a remote nginx-rtmp

nginx.conf:

rtmp {

server {

listen 1935;

application nanit {

live on;



pull rtmp://<nginx holding our stream>/internal/$name;

}

}

}

The internal part of the path here, in case you noticed and are wondering, is just another rtmp application used to pull the stream from.

This is what we‘re going to do:

Launch a Consul cluster using Helm

(and have it with a fixed ip address)

(and have it with a fixed ip address) Configure kube-dns to use Consul as a custom DNS (stub domains)

For every stream published to Nginx, create a DNS record

When playing a stream, use its unique address that will be resolved by our DNS to the right IP address of the Nginx that is holding it

Launching a Consul cluster using Helm

This is quite simple and straightforward. We are going to use the stable Consul Helm chart found here with its default values. Please note that Consul’s default DNS port is 8600 and is defined by the value ConsulDnsPort.

helm install --name consul stable/consul

This should create a release named consul with a high availability Consul cluster. Nice!

Now lets create a service to point to the cluster with a fixed IP to be used as our custom DNS service:

apiVersion: v1

kind: Service

metadata:

annotations:

service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"

labels:

chart: consul-1.2.1

component: consul-consul

heritage: Tiller

release: consul

name: consul-dns

namespace: default

spec:

clusterIP: 100.64.0.11

ports:

- name: consuldns-tcp

port: 53

protocol: TCP

targetPort: 8600

- name: consuldns-udp

port: 53

protocol: UDP

targetPort: 8600

selector:

component: consul-consul

sessionAffinity: None

type: ClusterIP

Note that we are using 100.64.0.11 as our fixed DNS IP address and we are pointing port 53 (default DNS port) to Consul’s default DNS port 8600.

Configuring kube-dns to use Consul as custom DNS

We will create a ConfigMap to define a stub domain that will route DNS requests that end with consul to our Consul DNS service:

apiVersion: v1

kind: ConfigMap

metadata:

name: kube-dns

namespace: kube-system

data:

stubDomains: |

{"consul": ["100.64.0.11"]}

Note the fixed IP of our Consul cluster (100.64.0.11)

Now if we query kube-dns for anything of the form something.service.consul it will be routed to our Consul DNS service.

Registering our stream as a service in Consul

Now that we have our custom DNS service ready to be used, we want to create a DNS record for every stream published to our cluster. We will do that in our nginx.conf file.

Before we go on with nginx.conf, it’s important to understand how Consul works and specifically the difference between registering a service directly to the Catalog vs registering with an Agent. I won’t go into details about Consul here, but I can tell you that we would want to use the Catalog for registering since a stream can disconnect and reconnect to a different nginx-rtmp instance and we would want to override the DNS record, while if we were using an Agent to register the service we would have ended up with multiple IP addresses per stream.

nginx.conf

We will use nginx-rtmp’s exec_publish and Consul’s HTTP API to register the stream directly in the Catalog:

rtmp {

... server {

...



application nanit {

live on;



exec_publish curl -XPUT -H "Content-Type: application/json" http://consul-consul:8500/v1/catalog/register -d '{"node":"dns", "address":"${LOCAL_IP}", "service": {"service":"$name", "address":"${LOCAL_IP}"}}';

}

}

}

exec_publish is called by the nginx-rtmp module once when a stream is published to the nanit application.

We use curl to make an HTTP API call to our Consul service with a fictitious node name dns. We use a fictitious node name since we are registering directly to the Catalog, and if we were to put a real node here and that node would have died at some point, then we would have ended up with missing services. This fictitious node does not have a health check associated with it so it can never “die”.

LOCAL_IP is an environment variable that holds the IP address of this nginx’s instance. We pass it in the Kubernetes pod definition and use a bash script in the container’s entrypoint to replace it with its value in the nginx.conf file. Sorry for not going into this, since its not the point of our story here.

$name will be the name of our service in the Consul catalog. This means we will be able to resolve its IP address like so: $name.service.consul

Querying our DNS service directly from Nginx

At this point we have a Consul cluster holding DNS records for each and every stream that is published to our cluster. Now all that is left is to use it when we want to pull a stream from the Nginx instance that holds it.

rtmp {

server {

listen 1935;

application nanit {

live on;



pull rtmp://$name.service.consul/internal/$name;

}

}

}

$name.service.consul will be resolved by the DNS to the IP address of the Nginx that holds the stream.

Summary

That’s it! We set up a custom DNS service using Consul, configured kube-dns to use it on a custom stub domain and we can now query for a stream from everywhere in our cluster, making our streams first-class citizens of our Kubernetes cluster.

At Nanit we have other services that use DNS to resolve a stream’s IP address and find the specific pod that handles it. Consul can be easily scaled up and has great performance.

I hope this post has helped you with whatever you are looking for. Don’t hesitate to let us know what you think in the comments!