Hey,

with the upcoming release of HAProxy 1.8 (see the blog post at haproxy.com) it’ll be possible to keep your stack behind the goodness of http2 without changing your code at all.

That’s pretty cool as you can have very perceptive differences under real-life scenarios. One thing to notice is that browsers only establish these connections if you’re HTTPS ready, and that means having TLS certificates in your load-balancer (or regular server).

Here are my 2 cents on how you can have a fully functioning HAProxy set up with certificate generation via Letsencrypt.

Dependencies

There are a handful of dependencies that need to be in place: certbot , lua , openssl-dev , zlib-dev , libpcre-dev and haproxy itself. If you already have all of those set, just skip to the next session: HAProxy Configuration section.

Here I’m making use of Ubuntu zesty (17.04) so there might be some differences between what I document here and your OS specifics.

Certbot

Start by adding certbot 's apt repository:

# after issuing the command press `enter` to accept sudo add-apt-repository ppa:certbot/certbot This is the PPA for packages prepared by Debian Let's Encrypt Team and backported for Ubuntu(s). More info: https://launchpad.net/~certbot/+archive/ubuntu/certbot Press [ENTER] to continue or ctrl-c to cancel adding it gpg: keybox '/tmp/tmp028qrfr4/pubring.gpg' created gpg: /tmp/tmp028qrfr4/trustdb.gpg: trustdb created gpg: key 8C47BE8E75BCA694: public key "Launchpad PPA for certbot" imported gpg: Total number processed: 1 gpg: imported: 1 OK

then update:

sudo apt update -y Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu zesty InRelease Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu zesty-updates InRelease [89.2 kB] Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu zesty-backports InRelease [89.2 kB] Get:4 http://ppa.launchpad.net/certbot/certbot/ubuntu zesty InRelease [21.3 kB] Hit:5 http://security.ubuntu.com/ubuntu zesty-security InRelease ...

and now install the desired package, python-certbot :

sudo apt install -y python-certbot Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: certbot dialog python-acme python-asn1crypto python-certifi python-chardet python-configargparse python-configobj python-cryptography python-dialog ... certbot help ------------------------------------------------------------- certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ... Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. The most common SUBCOMMANDS and flags are: ...

Once we have that we know that certbot is ready to request certificates from Letsencrypt for us.

HAProxy

Before installing HAProxy itself we need to get its dependencies right. HAProxy by itself can’t serve content from a directory like a static hosting web server would so to do that we must make use of lua which allows us to very easily extend HAProxy functionality. Then, the first dependency we’ll get is lua .

# set the version of Lua that we want to install. # is a variable that we can reference later. LUA_VERSION = 5.3.3 # install a development library that lua depends on sudo apt install -y \ libreadline-dev # fetch lua's source code for the version we want curl \ -SOL https://www.lua.org/ftp/lua- $LUA_VERSION .tar.gz # create a directory to hold that source code sudo mkdir \ -p /usr/src/lua # extract the source code to the directory we want sudo tar \ -xzf lua- $LUA_VERSION .tar.gz \ -C /usr/src/lua --strip-components = 1 # build Lua using a concurrency equivalent to the number # of processors we have (and targetting Linux) sudo make \ -C /usr/src/lua \ -j " $( getconf _NPROCESSORS_ONLN ) " \ linux # install it so that it's widely accessible sudo make \ -C /usr/src/lua \ install

To proceed with the HAProxy installation we need to know where the lua headers and compiled library are:

# search for the headers find /usr/local/include -name "*lua*" /usr/local/include/lua.h /usr/local/include/lualib.h /usr/local/include/lua.hpp /usr/local/include/luaconf.h # search for the lib find /usr/local/lib -name "*lua*" /usr/local/lib/liblua.a /usr/local/lib/lua

The next dependencies can all be fetched from apt though. We need three:

OpenSSL : for full TLS support (e.g, SNI extension);

: for full TLS support (e.g, extension); PCRE : to not rely on libc’s regex support but the more common Perl regex

: to not rely on libc’s regex support but the more common Perl regex ZLIB : for providing deflate and gzip compression algorithms

sudo apt install -y \ libpcre3-dev \ libssl-dev \ zlib1g-dev Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: ...

Now that our dependencies are all ready to be used we can proceed with HAProxy itself.

For obtaining the source code we can head to the downloads page (haproxy.org/download/1.8/src/) and fetch the latest version (I’m getting the last release candidate of 1.8 because this version gives us HTTP2 support, which is something I plan to write about soon).

# fetch the .tar.gz of the source code of haproxy curl -SOL http://www.haproxy.org/download/1.8/src/haproxy-1.8-rc3.tar.gz # decompress it to the current directory tar xzf ./haproxy-1.8-rc3.tar.gz

Once everything is in place it’s just a matter of compiling it with some flags that signalize the build process what is our environment and where it can find the Lua dependency:

# get into the source code directory cd ./haproxy-1.8-rc3/ # build the code with Lua, gzip compression, PCRE regex # targetting a recent kernel. make \ TARGET = linux2628 \ USE_OPENSSL = 1 \ USE_ZLIB = 1 \ USE_PCRE = 1 \ USE_LUA = 1 \ LUA_LIB_NAME = lua \ LUA_LIB = /usr/local/lib/ \ LUA_INC = /usr/local/include # make it available from `$PATH` so that # we can call `haproxy` from anywhere and # also have `man` pages set sudo make install

If you wonder why TARGET=linux2628 , head to the Makefile at haproxy 's source and check this:

ifeq ( $( TARGET ) ,linux2628) # This is for standard Linux >= 2.6.28 with # netfilter, epoll, tproxy and splice. USE_NETFILTER = implicit USE_POLL = implicit USE_EPOLL = implicit USE_TPROXY = implicit USE_LIBCRYPT = implicit USE_LINUX_SPLICE = implicit USE_LINUX_TPROXY = implicit USE_ACCEPT4 = implicit USE_FUTEX = implicit USE_CPU_AFFINITY = implicit ASSUME_SPLICE_WORKS = implicit USE_DL = implicit USE_THREAD = implicit else

In summary, it means that it’ll use some recent kernel capabilities to improve our performance.

After the compilation finishes you should have something like this:

./haproxy -vvvv HA-Proxy version 1.8-rc3-34650d5 2017/11/11 Copyright 2000-2017 Willy Tarreau <willy@haproxy.org> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-null-dereference -Wno-unused-label OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016 Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 Built with Lua version : Lua 5.3.3 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with network namespace support. Built with zlib version : 1.2.11 Running on zlib version : 1.2.11 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Encrypted password support via crypt(3): yes Built with PCRE version : 8.39 2016-06-14 Running on PCRE version : 8.39 2016-06-14 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with multi-threading support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available filters : [SPOE] spoe [COMP] compression [TRACE] trace

From the output, we can make sure that what we wanted has been properly compiled. Now we can move to the HAProxy configuration.

HAProxy Configuration

With all the dependencies installed we can proceed with the actual HAProxy configuration.

Because we need to have HAProxy performing something that it can’t do by default as mentioned - serve static files from a directory - we need to add a Lua plugin that does the job. That’s needed to serve the challenge that letsencrypt gives us when checking if we can serve content from a given domain (webroot).

Thanks to Jan (github.com/janeczku), we don’t have to write our own:

# get out of the haproxy directory cd ../ # fetch the lua script code that will serve the challenges # placed by certbot at a specific directory (webroot) git clone https://github.com/janeczku/haproxy-acme-validation-plugin # go to the repository directory and check what are the # files that we have there cd ./haproxy-acme-validation-plugin tree . ├── LICENSE ├── README.md ├── acme-http01-webroot.lua ├── cert-renewal-haproxy.sh └── haproxy.cfg.example

In the acme-http01-webroot.lua make sure you set the non_chroot_webroot variable to a location where we’ll set certbot to put challenges on:

-- -- Configuration -- -- When HAProxy is *not* configured with the 'chroot' option you must set an absolute path here and pass -- that as 'webroot-path' to the letsencrypt client acme.conf = { [ "non_chroot_webroot" ] = "/tmp/webroot" }

With the Lua plugin in place all we need to do next is specify that path to the plugin in our haproxy.cfg :

# notice that we're specifying in # `lua-load` the location of the Lua # script. Make sure you reference it # according to your configuration. global maxconn 8192 log 127.0.0.1 local0 tune.maxrewrite 16384 tune.bufsize 32768 tune.ssl.default-dh-param 2048 max-spread-checks 200 spread-checks 5 lua-load /home/ubuntu/haproxy-acme-validation-plugin/acme-http01-webroot.lua # Configure some default values that # frontends and backends can inherit # Here I'm setting some dummy timeouts. # In another blog post we can go through # some real values there. defaults log global retries 3 option redispatch option dontlog-normal mode http timeout http-request 10m timeout client 10m timeout connect 10m timeout server 10m timeout http-keep-alive 10m timeout tunnel 10m timeout client-fin 10m timeout server-fin 10m # The HTTP frontend serving only as a way of redirecting traffic # to port :443 where it can serve the real content via HTTPS # and also accept the requests from letsencrypt. # The big thing here is `use-service` which essentially means that # when the `url_acme_http01` ACL is `true` it'll execute the lua # script that we registered. # The name `lua.acme-http01` comes from the lua script itself: # # core.register_service("acme-http01", "http", acme.http01) # frontend http bind *:80 acl url_acme_http01 path_beg /.well-known/acme-challenge/ http-request use-service lua.acme-http01 if METH_GET url_acme_http01 redirect scheme https if METH_GET !url_acme_http01 # Serve the HTTPS traffic. # For each request that comes it takes the server name indicated # in SNI (the TLS extension that gives to an encrypted connection the # equivalent of a Host header) it looks on the list of certificates # that it loaded from the `crt-list` file and then uses that certificate. # As we're offloading the TLS resolution from the backend to the # frontend, `backend_test` will receive unencrypted content as # if there was a plain TCP connection coming (no need to deal with # certificates there - only HAProxy has to care about it). frontend https bind *:443 default_backend backend_test # Dummy backend connecting haproxy to a server on the same machine. # The server at `localhost:8080` is a simple HTTP server expecting # regular non-encrypted connections. backend backend_test server test-server localhost:8080

To make sure we got our configuration right we can make use of the -c flag of HAProxy to perform some sort of “dry run” and only check the config. Assuming we have the configuration at ~/haproxy.cfg :

haproxy --help ... Usage : haproxy -f <cfgfile | cfgdir> ] [ opt ] ... -c check mode : only check config files and exit ... haproxy -c -f ~/haproxy.cfg Configuration file is valid

As the configuration is valid we can start HAProxy with a privileged user (as it’s going to bind on ports 80 and 443):

sudo haproxy -f ./haproxy.cfg [ info ] 328/123916 ( 374961 ) : [ acme ] http-01 plugin v0.1.1

Now that we got HAProxy working we can check if the lua plugin is really receiving the requests when we hit port 80 on the expected path:

# in a separate terminal, make a request to the # haproxy instance on the path which should respond # to letsencrypt requests curl localhost:80/.well-known/acme-challenge/aa resource not found # on the HAProxy terminal, check the logs [ warning ] 328/124202 ( 374961 ) : [ acme ] http-01 token not found: aa ( client-ip: 127.0.0.1 )

As it’s all working as expected we can request a certificate to letsencrypt.

# Create the directory where challenges will # be placed. This is the directory that you have # to be set earlier in the lua script in the # `acme.conf` field. mkdir /tmp/webroot # make certbot initiate the certificate generation # process using the webroot method, placing challenges # at the directory /tmp/webroot and creating the certificate # for the domain `cirocosta.com` sudo certbot certonly --webroot -w /tmp/webroot -d cirocosta.com Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator webroot, Installer None ... - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/cirocosta.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/cirocosta.com/privkey.pem Your cert will expire on 2018-02-23. To obtain a new or tweaked ...

With the certificates in hand now we have to create a new file that HAProxy can use to terminate the TLS connections. This new file is a concatenation of the private key and the certificate we received from letsencrypt:

# concatenate those files (as I used the root user to # provision them - via certbot - they were created as # root, so I must be `root` to see them). sudo cat \ /etc/letsencrypt/live/cirocosta.com/privkey.pem \ /etc/letsencrypt/live/cirocosta.com/fullchain.pem > \ ~/cirocosta.com

Then update the haproxy.cfg file to make use of the certificate when bind ing to port 443 :

frontend https # bind *:443 (before) bind *:443 ssl crt /home/ubuntu/cirocosta.com default_backend backend_test

Then perform a soft-reload HAProxy after checking if the configuration is fine (it checks the cert):

# check the configuration haproxy -c -f ~/haproxy.cfg Configuration file is valid # soft-reload the current instance by creating a new # one passing the pid of the old instance via the `-sf` # flag sudo haproxy \ -f ./haproxy.cfg \ -sf $( pidof haproxy ) [ info ] 328/131241 ( 376196 ) : [ acme ] http-01 plugin v0.1.1 # in the terminal where the old HAProxy was running # you should see the following (perfectly fine) messages: [ WARNING ] 328/131241 ( 374961 ) : Stopping frontend http in 0 ms. [ WARNING ] 328/131241 ( 374961 ) : Stopping frontend https in 0 ms. [ WARNING ] 328/131241 ( 374961 ) : Stopping backend backend_test in 0 ms. [ WARNING ] 328/131241 ( 374961 ) : Proxy http stopped ( FE: 3 conns, BE: 0 conns ) . [ WARNING ] 328/131241 ( 374961 ) : Proxy https stopped ( FE: 0 conns, BE: 0 conns ) . [ WARNING ] 328/131241 ( 374961 ) : Proxy backend_test stopped ( FE: 0 conns, BE: 0 conns ) .

And that’s it!

Even without a properly configured server, we should already be able to see that the HTTPS setup is fine:

Closing thoughts

Even though the guide is quite extensive if you consider the process of building HAProxy and Lua from scratch, there’s not much going on:

Make HAProxy serve the challenges at port 80 on a specific path that letsencrypt expects Concatenate certificates and private keys Update the HAProxy configuration Reload HAProxy

Once that’s automated you’re ready to serve your customers with HTTPS / TLS without dropping connections and having to pay for certificates.

Even if the part of setting up the dependencies seem like too much you can tailor a Dockerfile that does all of that and you’re good to go.

If you enjoyed the guide and are willing to learn more about related content, make sure you subscribe to the newsletter. In case you find mistakes I made or think there’s something to improve, just let me know, I’m cirowrc on Twitter.

Have a good one!

finis

If you’re interested in provisioning TLS certificates, you’re probably also interested in HTTP/2.

I just finished writing a blog post on how to make use of HTTP/2 Server Push with NGINX for those out there making use of NGINX.

Have a good one!