haproxy-auth-request ¶ HTTP access control using subrequests – 2018-01-19

Direct link to the repository for the impatient.

For a project of mine I needed to authenticate a medium number of vHosts behind an haproxy to the same group of users. I usually use TLS client certificates to protect internal services only I (or a small number of technically savy people) need to access. But they come with some serious issues in usablity, so I quickly disregarded them. Another popular choice is HTTP authentication (“ .htaccess ”), but that would require setting up and securely distributing credentials to the users that need to access the vHosts. On top of that the haproxy documentation discourages using hashed passwords, because the hash algorithms are designed to be CPU intensive and thus would block haproxy's event loop. I don't want to store unhashed passwords, even if they are coming from a secure random number generator. Setting up IP based protection or a VPN was out of question as well, because it either is insecure or too hard to set up for the target group of users.

Some time ago I learned about bitly/oauth2_proxy which does exactly what it's name implies: It verifies users against some OAuth 2 service, sets a cookie and tunnels all the requests to a configured upstream if the cookie is valid. As all the users already are part of an OAuth 2 provider this seemed to fit my needs, except for one thing: I did not want to send everything from the Internet, via haproxy, via oauth2_proxy to the backend services for these reasons:

Configuring different upstreams is based on the path according to the README, I needed to route based on the Host. It does not allow for load balancing between multiple backends. I would have to put a load balancer behind oauth2_proxy, leading to even more possible points of failure.

Luckily oauth2_proxy also supports an endpoint that just returns whether a request should be allowed or not: I would be able to ask oauth2_proxy whether the request is good and perform the remaining delegation in haproxy. There is only one issue: haproxy supports nothing like nginx' auth_request. But it supports Lua - it should be possible to build something.

Meet Lua ¶ Disclaimer: This post was written after the fact and thus the actual timeline of things does not match up. Partly to allow for an easier explanation, partly because I don't remember the actual progression of my Lua script. All the code snippets in this post are licensed under the terms of the MIT license. It was not my first time programming Lua, I built two small Lua programs into nginx before. It was my first time programming Lua in haproxy though. The first thing was to learn how exactly Lua was being called from haproxy: The first Lua program ¶ I searched for the documentation of Lua in haproxy and started reading. There are four main things I could register into haproxy: action Allows to modify requests. converters Allows to modify samples. fetches Allows to add samples. service Allows to answer requests. At first I misunderstood what a service does and tried to use that one to handle the auth requests. But as I needed to modify requests and then pass them to a backend I needed an action: auth-request.lua core.register_action("auth-request", { "http-req" }, function(txn) -- code here end) haproxy.cfg global lua-load auth-request.lua frontend http mode http option httplog bind :::80 v4v6 http-request lua.auth-request haproxy would now call my function for each HTTP request. While looking around the Lua documentation I found the print_r function, which would make debugging easier, as I did not have to put the information into response headers. I promptly added it to my script. When running haproxy using the -d option it would print the parameter to my console. The first HTTP request ¶ Next thing was searching for a Lua HTTP library, I really did not want to implement HTTP on my own. I quickly found socket.http and installed it using apt-get install lua-socket . A quick test proofed successful: I was able to make HTTP requests from within my function: auth-request.lua core.register_action("auth-request", { "http-req" }, function(txn) local b, c, h = http.request { url = "http://127.0.0.1" } print_r(b) print_r(c) end) haproxy's Sockets ¶ I however noticed the Socket class in haproxy's documentation before and thought that there must be a reason why it exists. Possibly the default Socket class would block haproxy's event loop. Luckily the developers of socket.http imagined that use case and support passing a custom constructor as create to the http.request function. I quickly added it and tried it out: Nothing happened. An strace confirmed that haproxy was not even connecting to the host. Poking around in the http.socket module revealed that commenting out the call to settimeout would allow it to set up the socket. I figured I would find out what haproxy's Socket is doing differently later: auth-request.lua core.register_action("auth-request", { "http-req" }, function(txn) local b, c, h = http.request { url = "http://127.0.0.1", create = core.tcp } print_r(b) print_r(c) end) Now haproxy was sending the HTTP request and even reading the response. First success! Unfortunately instead of the response code it now was printing: connection closed. to my console. By ag ing that message in haproxy's source code I could find that the issue must be somewhere inside the receive method. print_r ing the results of the receive calls in socket.http lead me to the receiveheaders function. For some reason that line was returning an empty string instead of a valid HTTP header, it did however read the HTTP status line in receivestatusline successfully before. I figured to explicitly add the default parameter of "*l" to instruct it to read a line. With success. haproxy must be incorrectly handling the default parameter, despite explicitely documenting that no parameter equals *l . As I did not want to patch all of http.lua I tried to find out whether I could monkey patch the Socket class inside of Lua: auth-request.lua function create_sock() local sock = core.tcp() sock.old_receive = sock.receive sock.receive = function(socket, pattern, prefix) local a, b if pattern == nil then pattern = "*l" end if prefix == nil then a, b = sock:old_receive(pattern) else a, b = sock:old_receive(pattern, prefix) end return a, b end return sock end core.register_action("auth-request", { "http-req" }, function(txn) local b, c, h = http.request { url = "http://127.0.0.1", create = create_sock } print_r(b) print_r(c) end) It now correctly printed the response code. I was able to make HTTP requests with haproxy's Socket class. On to find out why the call to settimeout inside socket.http caused the request to fail, despite settimeout working from within my Lua code. Poking around in the source code of socket and haproxy revealed that haproxy was returning 0 , while the original socket class was returning 1 . For my workstation I fixed the return code in haproxy and recompiled it. With success. For production I did not want to run a self compiled haproxy and thus monkey patched settimeout as well: auth-request.lua function create_sock() local sock = core.tcp() sock.old_receive = sock.receive sock.receive = function(socket, pattern, prefix) local a, b if pattern == nil then pattern = "*l" end if prefix == nil then a, b = sock:old_receive(pattern) else a, b = sock:old_receive(pattern, prefix) end return a, b end sock.old_settimeout = sock.settimeout sock.settimeout = function(socket, timeout) socket:old_settimeout(timeout) return 1 end return sock end core.register_action("auth-request", { "http-req" }, function(txn) local b, c, h = http.request { url = "http://127.0.0.1", create = create_sock } print_r(b) print_r(c) end) The actual patch in haproxy probably occured later, as I fixed the bug in the receive function before fixing settimeout . I fixed IPv6 support first as well. Making DNS requests ¶ Lua in haproxy is documented to not support DNS requests. I did not want to hardcode an IP address in my configuration, however. While searching through haproxy's Lua documentation I noticed that I could get information about the backends using core.backends , with the Server class carrying a get_addr method. Would that give me the resolved IP address? It turned out that it does: haproxy.cfg global lua-load auth-request.lua frontend fe mode http bind :::8080 v4v6 http-request lua.auth-request default_backend be backend be mode http server s example.com:80 auth-request.lua core.register_action("auth-request", { "http-req" }, function(txn) local b, c, h = http.request { url = "http://" .. core.backends["be"].servers["s"]:get_addr(), create = create_sock } print_r(b) print_r(c) end) Better! While the name of the backend still is hardcoded the IP address is not. At first I used the txn:get_var("txn.foo") method to read a variable I set using http-request set-var("txn.foo") string("be") , but while making the bug fixes to haproxy I noticed that I was able to pass additional parameters to my Lua action: haproxy.cfg global lua-load auth-request.lua frontend fe mode http bind :::8080 v4v6 http-request lua.auth-request be default_backend be backend be mode http server s example.com:80 auth-request.lua core.register_action("auth-request", { "http-req" }, function(txn, be) local b, c, h = http.request { url = "http://" .. core.backends[be].servers["s"]:get_addr(), create = create_sock } print_r(b) print_r(c) end, 1) This later evolved into looping through all the servers in that backend and checking their health status. I am now able to make HTTP requests to some easily changed backend. I now needed to return the response information back to haproxy: Telling haproxy the result of the auth-request ¶ Similarly to my first attempt of passing the backend name I could pass variables back to haproxy using txn:set_var("txn.foo", true) . This was fairly straight forward: auth-request.lua core.register_action("auth-request", { "http-req" }, function(txn, be) txn:set_var("txn.auth_response_successful", false) local b, c, h = http.request { url = "http://" .. core.backends[be].servers["s"]:get_addr(), create = create_sock } if 200 <= c and c < 300 then txn:set_var("txn.auth_response_successful", true) end end, 1) Passing request headers ¶ The only thing missing is passing all the request headers to my auth service (as it needed to know whether a specific cookie is set or not). This required me to convert from haproxy's header representation to the one of socket.http : haproxy passes header values as a table to support duplicate headers. socket.http only supports a single header name to string mapping. Luckily RFC 7230 allows me to support this without losing information: A recipient MAY combine multiple header fields with the same field name into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field value to the combined field value in order, separated by a comma. The order in which header fields with the same field name are received is therefore significant to the interpretation of the combined field value; a proxy MUST NOT change the order of these field values when forwarding a message. auth-request.lua core.register_action("auth-request", { "http-req" }, function(txn, be) txn:set_var("txn.auth_response_successful", false) local headers = {} for header, values in pairs(txn.http:req_get_headers()) do for i, v in pairs(values) do if headers[header] == nil then headers[header] = v else headers[header] = headers[header] .. ", " .. v end end end local b, c, h = http.request { url = "http://" .. core.backends[be].servers["s"]:get_addr(), create = create_sock, headers = headers } if 200 <= c and c < 300 then txn:set_var("txn.auth_response_successful", true) end end, 1)