Securing an HTTPS server

In response to numerous comments about "excessive minimalism", I recently put together a new website for my Tarsnap online backup service ; and since I was reworking things anyway, I decided that it was a good time to move to a new web server and generally clean up the system configuration. Among the things I cleaned up was how I handle HTTPS: I need it because people enter passwords when creating tarsnap accounts and when logging in to the tarsnap account management interface, but I wasn't satisfied with the (in)security of running Apache with SSL enabled.

There are two reasons why I don't like the idea of running Apache with mod_ssl and OpenSSL to service HTTPS requests. First, SSL is a complex protocol, and thus implementations are likely to be buggy and insecure; and second, SSL certificates are sensitive information. If someone finds a vulnerability in OpenSSL, I don't want them to be able to leverage that into accessing anything else on the server; and likewise, if someone finds a vulnerability in something else (Apache, for instance, or -- although it would greatly shock me -- one of the tarsnap CGI scripts) I don't want them to be able to leverage that into gaining access to the SSL certificate for www.tarsnap.com.

In order to maximize the separation between systems on this machine, I set it up following the "service jail" model: Using FreeBSD's jail functionality (roughly speaking, chroot redesigned to be secure), I have daemons running each in their own "world" and bound to individual non-routable IP addresses; and then I use the ipfw firewall and natd daemon to translate and pass packets back and forth to the external interface. Among the jails on this system, I have a package-building jail on IP address 192.168.128.1; a dns cache jail on IP address 192.168.128.53; an outgoing mail daemon on IP address 192.168.1.25; and an authoritative nameserver on IP address 192.168.0.53.

TCP connections incoming to port 80 on www.tarsnap.com are redirected by natd to an Apache process running on IP address 192.168.0.80, while incoming connections to port 443 -- that being the standard port for HTTPS traffic -- is redirected to IP address 192.168.0.44. There they encounter the stunnel daemon, which is running with the following configuration file:

chroot = /var/run/stunnel pid = /stunnel.pid setgid = stunnel setuid = stunnel output = /var/log/stunnel.log syslog = no [https] accept = 192.168.0.44:443 connect = 192.168.0.80:80 local = 192.168.0.44 cert = /usr/local/etc/ssl/server.crt key = /usr/local/etc/ssl/server.key

As one might expect from the configuration file, stunnel takes the incoming HTTPS connections, performs the necessary SSL negotiation, and then passes the underlying HTTP connection on to the Apache daemon at 192.168.0.80. Now if someone compromises OpenSSL, they will still be stuck within the stunnel jail -- able to steal the SSL certificate, to be sure, and also able to intercept other HTTPS connections -- but unable to do anything to Apache (or the CGI scripts which Apache launches) aside from what could already be done from the outside world.

There is one slight inconvenience to this configuration, however: Because HTTPS connections are proxied via stunnel, Apache's logs show stunnel's IP address as the source of each connection, with obvious consequences for log analysis. That said, there is a workaround available: Apache 2.2 can be instructed to log source port numbers, and stunnel logs all connections it handles -- including the source IP address of the incoming connection and the port number used for opening up the connection to Apache. Consequently, it is possible to correlate Apache and stunnel's logs in order to generate a traditional HTTP log file including "real" source addresses. (I might blog further about this in the future -- if you're interested, please leave a comment below.)

Good system security requires multiple layers. It isn't enough to use software, like Apache, mod_ssl, and OpenSSL, which have no presently known vulnerabilities, and upgrade them whenever vulnerabilities are found. Designing secure systems requries accepting that security vulnerabilities will be found -- and exploited before they can be fixed -- and putting components together in such a way as to minimize the potential impact of such vulnerabilities.

Disqus