What’s New in HAProxy 1.6

[ANNOUNCE] HAProxy 1.6.0 released

Yesterday, 13th of October, Willy has announced the release of HAProxy 1.6.0, after 16 months of development!

First good news is that release cycle goes a bit faster and we aim to carry on making it as fast.

A total of 1156 commits from 59 people were committed since the release of 1.5.0, 16 months ago.

Please find here the official announce: [ANNOUNCE] haproxy-1.6.0 now released!.

In his mail, Willy detailed all the features that have been added to this release. The purpose of this blog article is to highlight a few of them, providing the benefits and some configuration examples.

NOTE: Most of the features below were already backported and integrated into our HAProxy Enterprise and ALOHA products.

HAProxy Enterprise is our Open Source version of HAProxy based on HAProxy community stable branch where we backport many features from dev branch and we package it to make the most stable, reliable, advanced and secured version of HAProxy. It also comes with third party software to fill the gap between a simple HAProxy process and a load-balancer (VRRP, syslog, SNMP, Route Health Injection, etc…). Cherry on the cake, we provide “enterprise” support on top of it.

NOTE 2: the list of new features introduced here is not exhaustive. Example are proposed in a quick and dirty way to teach you how to start with the feature. Don’t run those examples in production 🙂

It’s 2015, let’s use QUOTE in configuration file

Those who uses HAProxy for a long time will be happy to know that ‘\ ‘ (backslash-space) sequence is an old painful souvenir with 1.6 🙂

We can now write:



reqirep "^Host: www.(.*)" "Host: foobar\1"

or

option httpchk GET / "HTTP/1.1\r

Host: www.domain.com\r

Connection: close"

Lua Scripting

Maybe the biggest change that occurred is the integration of Lua.

Quote from Lua’s website: “Lua is a powerful, fast, lightweight, embeddable scripting language.“.

Basically, everyone has now the ability to extend HAProxy by writing and running their own Lua scripts. No need to write C code, maintain patches, etc…

If some Lua snipplets are very popular, we may write the equivalent feature in C and make it available in HAProxy mainline.

One of the biggest challenge Thierry faced when integrating Lua is to give it the ability to propose non-blocking processing of Lua code and non-blocking network socket management.

HAProxy requires Lua 5.3 or above.

With Lua, we can add new functions to the following HAProxy elements:

Service

Action

Sample-fetch

Converter

Compiling HAProxy and Lua

Installing LUA 5.3

cd /usr/src curl -R -O http://www.lua.org/ftp/lua-5.3.0.tar.gz tar zxf lua-5.3.0.tar.gz cd lua-5.3.0 make linux sudo make INSTALL_TOP=/opt/lua53 install

LUA 5.3 library and include files are now installed in /opt/lua53.

Compiling HAProxy with Lua support

make TARGET=linux2628 USE_OPENSSL=1 USE_PCRE=1 USE_LUA=1 LUA_LIB=/opt/lua53/lib/ LUA_INC=/opt/lua53/include/

HAProxy/Lua simple Hello world example

A simple Hello world! in Lua could be written like this:

The lua code in a file called hello_world.lua :

core.register_service("hello_world", "tcp", function(applet) applet:send("hello world

") end)

: The haproxy configuration

global lua-load hello_world.lualisten proxy bind 127.0.0.1:10001 tcp-request content use-service lua.hello_world

More with Lua

Please read the doc, ask your questions on HAProxy‘s ML (no registration needed): haproxy@formilux.org.

Of course, we’ll write more articles later on this blog about Lua integration.

Captures

HAProxy‘s running context is very important when writing configuration. In HAProxy, each context is isolated. IE: you can’t use a request header when processing the response.

With HAProxy 1.6, this is now possible: you can declare capture slots, store data in it and use it at any time during a session.



defaults mode http frontend f_myapp bind :9001 declare capture request len 32 # id=0 to store Host header declare capture request len 64 # id=1 to store User-Agent header http-request capture req.hdr(Host) id 0 http-request capture req.hdr(User-Agent) id 1 default_backend b_myapp backend b_myapp http-response set-header Your-Host %[capture.req.hdr(0)] http-response set-header Your-User-Agent %[capture.req.hdr(1)] server s1 10.0.0.3:4444 check

Two new headers are inserted in the response:

Your-Host: 127.0.0.1:9001

Your-User-Agent: curl/7.44.0

Multiprocess, peers and stick-tables

In 1.5, we introduced “peers” to synchronize stick-table content between HAProxy servers. This feature was not compatible with multi-process mode.

We can now, in 1.6, synchronize content of the table it is stick to one process. It allows creating configurations for massive SSL processing pointing to a single backend sticked on a single process where we can use stick tables and synchronize its content.



peers article peer itchy 127.0.0.1:1023 global pidfile /tmp/haproxy.pid nbproc 3 defaults mode http frontend f_scalessl bind-process 1,2 bind :9001 ssl crt /home/bassmann/haproxy/ssl/server.pem default_backend bk_lo backend bk_lo bind-process 1,2 server f_myapp unix@/tmp/f_myapp send-proxy-v2 frontend f_myapp bind-process 3 bind unix@/tmp/f_myapp accept-proxy default_backend b_myapp backend b_myapp bind-process 3 stick-table type ip size 10k peers article stick on src server s1 10.0.0.3:4444 check

Log

log-tag

It is now possible to position a syslog tag per process, frontend or backend. Purpose is to ease the job of syslog servers when classifying logs.

If no log-tag is provided, the default value is the program name.

Example applied to the configuration snipplet right above:



frontend f_scalessl log-tag SSL [...] frontend f_myapp log-tag CLEAR [...]

New log format variables

New log format variables have appeared:

%HM: HTTP method (ex: POST)

%HP: HTTP request URI without query string (path)

%HQ: HTTP request URI query string (ex: ?bar=baz)

%HU: HTTP request URI (ex: /foo?bar=baz)

%HV: HTTP version (ex: HTTP/1.0)

Server IP resolution using DNS at runtime

In 1.5 and before, HAProxy performed DNS resolution when parsing configuration, in a synchronous mode and using the glibc (hence /etc/resolv.conf file).

Now, HAProxy can perform DNS resolution at runtime, in an asynchronous way and update server IP on the fly. This is very convenient in environment like Docker or Amazon Web Service where server IPs can be changed at any time.

Configuration example applied to docker. A dnsmasq is used as an interface between /etc/hosts file (where docker stores server IPs) and HAProxy:



resolvers docker nameserver dnsmasq 127.0.0.1:53 defaults mode http log global option httplog frontend f_myapp bind :80 default_backend b_myapp backend b_myapp server s1 nginx1:80 check resolvers docker resolve-prefer ipv4

(...) haproxy[15]: b_myapp/nginx1 changed its IP from 172.16.0.4 to 172.16.0.6 by docker/dnsmasq.

HTTP rules

Then, let’s restart s1 with the command “docker restart nginx1” and let’s have a look at the magic in the logs:

New HTTP rules have appeared:

http-request : capture, set-method, set-uri, set-map, set-var, track-scX, sc-inc-gpc0, sc-set-gpt0, silent-drop

: capture, set-method, set-uri, set-map, set-var, track-scX, sc-inc-gpc0, sc-set-gpt0, silent-drop http-response: capture, set-map, set-var, sc-inc-gpc0, sc-set-gpt0, silent-drop, redirect

Variables

We often used HTTP header fields to store temporary data in HAProxy. With 1.6, we can now define variables.

A variable is available for a scope: session, transaction (request or response), request, response.

A variable name is prefixed by its scope (sess, txn, req, res), a dot ‘.’ and a tag composed by ‘a-z’, ‘A-Z’, ‘0-9’ and ‘_’.

Let’s rewrite the capture example using variables



global # variables memory consumption, in bytes tune.vars.global-max-size 1048576 tune.vars.reqres-max-size 512 tune.vars.sess-max-size 2048 tune.vars.txn-max-size 256 defaults mode http frontend f_myapp bind :9001 http-request set-var(txn.host) req.hdr(Host) http-request set-var(txn.ua) req.hdr(User-Agent) default_backend b_myapp backend b_myapp http-response set-header Your-Host %[var(txn.host)] http-response set-header Your-User-Agent %[var(txn.ua)] server s1 10.0.0.3:4444 check

Mailers

Now, HAProxy can send emails when server states change (mainly goes DOWN), so your sysadmins/devos won’t sleep anymore :). It used to be able to log only before.



mailers mymailers mailer smtp1 192.168.0.1:587 mailer smtp2 192.168.0.2:587 backend mybackend mode tcp balance roundrobin email-alert mailers mymailers email-alert from test1@horms.org email-alert to test2@horms.org server srv1 192.168.0.30:80 server srv2 192.168.0.31:80

Processing of HTTP request body

Until 1.5 included, HAProxy could process only HTTP request headers. It can now access request body.

Simply enable the statement below in your frontend or backend to give HAProxy this ability:

option http-buffer-request

Protection against slow-POST attacks

slow-POST attacks are like the slowlorys one, except that HTTP header are sent quicly, but request body are sent very slowly.

Once enabled, the timeout http-request parameters also apply to the POSTED data.

Fetch methods

A few new fetch methods now exists to play with the body: req.body, req.body_param, req.body_len, req.body_size, etc…

Short example to detect “SELECT *” string in a request POST body, and of course how to deny it:



defaults mode http frontend f_mywaf bind :9001 option http-buffer-request http-request deny if { req.body -m reg "SELECT \*" } default_backend b_myapp backend b_myapp server s1 10.0.0.3:4444 check

New converters

1.5 introduced the converters but only a very few of them were available.

1.6 adds many of them. The list is too long, but let’s give the most important ones: json, in_table, field, reg_sub, table_* (to access counters from stick-tables), etc…

Device Identification

Through our company, we have some customer who want us to integrate into HAProxy the ability to detect device type and characteristics and report it to the backend server.

We got a couple of contributions from 2 companies experts in this domain: 51 degrees and deviceatlas.

You can now load those libraries in HAProxy in order to fully qualify a client capabilities and set up some headers your application server can rely on to adapt content delivered to the client or let the varnish cache server use it to cache multiple flavor of the same object based on client capabilities.

More on this blog later on how to integrate each product.

Seamless server states

Prior 1.6, when being reloaded, HAProxy considers all the servers are UP until the first check is performed.

Since 1.6, we can dump server states into a flat file right before performing the reload and let the new process know where the states are stored. That way, the old and new processes owns exactly the same server states (hence seamless).

The following information are reported:

server IP address when resolved by DNS

operational state (UP/DOWN/…)

administrative state (MAINT/DRAIN/…)

Weight (including slowstart relative weight)

health check status

rise / fall current counter

check state (ENABLED/PAUSED/…)

agent check state (ENABLED/PAUSED/…)

The state could be applied globally (all server found) or per backend.

Example:

Simple HAProxy configuration:



global stats socket /tmp/socket server-state-file /tmp/server_state backend bk load-server-state-from-file global server s1 10.0.0.3:4444 check weight 11 server s2 10.0.0.4:4444 check weight 12

Before reloading HAProxy, we save the server states using the following command:

socat /tmp/socket - <<< "show servers state" > /tmp/server_state

Here is the content of /tmp/server_state file:



1 # <field names skipped for the blog article> 1 bk 1 s1 10.0.0.3 2 0 11 11 4 6 3 4 6 0 0 1 bk 2 s2 10.0.0.4 2 0 12 12 4 6 3 4 6 0 0

Now, let’s proceed with reload as usual.

Of course, the best option is to export the server states using the init script.

External check

HAProxy can run a script to perform complicated health checks.

Just be aware about the security concerns when enabling this feature!!

Configuration:



global external-check backend b_myapp external-check path "/usr/bin:/bin" external-check command /bin/true server s1 10.0.0.3:4444 check

TLS / SSL

NOTE: some of the features introduced here may need a recent openssl library.

Detection of ECDSA-able clients

This has already been documented by Nenad on this blog: Serving ECC and RSA certificates on same IP with HAProxy

SSL certificate forgery on the fly

Since 1.6, HAProxy can forge SSL certificate on the fly!

Yes, you can use HAProxy with your company’s CA to inspect content.

Support of Certificate Transparency (RFC6962) TLS extension

When loading PEM files, HAProxy also checks for the presence of file at the same path suffixed by “.sctl”. If such file is found, support for Certificate Transparency (RFC6962) TLS extension is enabled.

The file must contain a valid Signed Certificate Timestamp List, as described in RFC. File is parsed to check basic syntax, but no signatures are verified.

TLS Tickets key load through stats socket

A new stats socket command is available to update TLS ticket keys at runtime. The new key is used for encryption/decryption while the old ones are used for decryption only.

Server side SNI

Many application servers now takes benefit from Server Name Indication TLS extension.

Example:



backend b_myapp_ssl mode http server s1 10.0.0.3:4444 check ssl sni req.hdr(Host)

Peers v2

The peers protocol, used to synchronized data from stick-tables between two HAProxy servers can now synchronise more than just the sample and the server-id.

It can synchronize all the tracked counters. Note that each node push its local counter to a peer. So this must be used for safe reload and server failover only.

Don’t expect to see 10 HAProxy servers to sync and aggregate counters in real time.

That said, this protocol has been extended to support different data type, so we may see more features soon relying on it 😉

HTTP connection sharing

On the road to HTTP/2, HAProxy must be able to support connection pooling.

And on the road to the connection pools, we have the ability to share a server side connection between multiple clients.

The server side connection may be used by multiple clients, until the owner (the client side connection) of this session dies. Then new connections may be established.

The new keyword is http-reuse and have different level of sharing connections:

never : no connections are shared

: no connections are shared safe : first request of each client is sent over its own connection, subsequent request may used an other connection. It works like a regular HTTP keepalive

: first request of each client is sent over its own connection, subsequent request may used an other connection. It works like a regular HTTP keepalive aggressive : send request to connections that has proved reliably support connection reuse (no quick connection close after a response has been sent)

: send request to connections that has proved reliably support connection reuse (no quick connection close after a response has been sent) always: send request to established connections, whatever happens. If the server was closing the connection in the mean time, the request is lost and the client must resend it.

Get rid of 408 in logs

Simply use the new option option http-ignore-probes .