Let’s talk about nginx-proxy & fleetctl

OK, so now you have your own cluster for CoreOS! I have only one machine, but it should work the same way. Next task is to deploy some containers, and the most important part: nginx-proxy. I’m the type of person who love using the terminal, but also some nice fancy web UI. The avantage of Docker is to deploy things easily, just to test for example a new fancy SaaS app. But I’m sick of writing some reverse-proxy stuff to have a nice and fancy URL like slides.pierrezemb.fr. So that’s where nginx-proxy comes in. It’ll watch for docker’s socket (ready-only mode), and generate the right conf file for nginx each time there’s some change ie a new container comes in the server. It handles HTTP/HTTPS and everything you need! We’ll now see how to deploy containers with it.

For the example, I’ll be using my own website. It’s available on the Docker Registry with some Automatic build stuff to automate the whole process of creation of the image. I’m using my own Web server: GoStatic. It’s a very light container (5MB) written in Go. So we need 2 containers: the proxy container and my portfolio-container. I’ll use a third container as a data container for my ssl certificates. I’ll use fleet, which is like systemd but in a distributed way.

From official CoreOS’s website

This is my service files. You can find all the files on my GitHub account, and some explication about systemd for Fleet config file here:

[Unit]

Description=Proxy

After=cert.service

Requires=cert.service [Service]

TimeoutStartSec=0

ExecStartPre=-/usr/bin/docker kill proxy

ExecStartPre=-/usr/bin/docker rm proxy

ExecStartPre=/usr/bin/docker pull jwilder/nginx-proxy:latest

ExecStart=/usr/bin/docker run — name proxy -p 80:80 -p 443:443 — volumes-from ssl-cert -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy

ExecStop=/usr/bin/docker stop proxy

Proxy Service

Nothing spectacular here, I’m listening to 443 and 80 to handle HTTPS/HTTP traffic. The volumes-from is because my ssl certificates are on another container, so I need to bind it together. Remove it and -p 443:443 if you’re not using SSL.

[Unit]

Description=Portfolio web server

After=proxy.service

Requires=docker.service

user=core

TimeoutStartSec=0

ExecStartPre=-/usr/bin/docker kill portfolio

ExecStartPre=-/usr/bin/docker rm portfolio

ExecStartPre=/usr/bin/docker pull pierrezemb/portfolio:latest

ExecStart=/usr/bin/docker run -e VIRTUAL_HOST=pierrezemb.fr,

ExecStop=/usr/bin/docker stop portfolio [Service]user=coreTimeoutStartSec=0ExecStartPre=-/usr/bin/docker kill portfolioExecStartPre=-/usr/bin/docker rm portfolioExecStartPre=/usr/bin/docker pull pierrezemb/portfolio:latestExecStart=/usr/bin/docker run -e VIRTUAL_HOST=pierrezemb.fr, www.pierrezemb.fr -p 8043:8043 — name portfolio pierrezemb/portfolio — forceHTTPExecStop=/usr/bin/docker stop portfolio

Service for my awesome website

Here, the options VIRTUAL_HOST allows the proxy to know the matching URL for this container. Our first container’ll see the docker run command, parse it, and write to good config file according to this options.

Before running this, you need to configure fleetctl. Add this to your .bashrc/.zshrc:

export FLEETCTL_TUNNEL=ip.of.coreos.machine

You can now run some fleetctl command! Additionnal tutorial about what fleetctl can do can be found here, but the basics are:

$ ssh-add ~/.ssh/coreos // Add the right SSH Key for fleetctl $ fleetctl list-machines // List all the machines

$ fleetctl start proxy.service // Start using proxy service

$ fleetctl start proxy.service // Start using portfolio service

This command will use SSH to connect, and the run the containers properly somewhere in your cluster. No need for SSH anymore. With this config, I can now scale easily my containers, and not being preoccupied by changing my reverse-proxy anymore. I need a new machine? I wrote a bash script, so it’s juste one command to add a new machine to my cluster. New container? i just need to write a fleet service, start it, update my DNS, and voilà.

I’m really happy with this stack, and it feels really easier than before.

What could be better?

I know that etcd could be a nice feature to use to register which container is using which URL, but I didn’t have the time to work on it yet. Maybe another time!