Nginx is by far my favourite Web server. After years going through various Apache flavors, Sun ONE, Microsoft IIS or Lighttpd twisted Perl like configuration, Nginx simplicity was a relief.

Last week, a friend of mine asked me for some help to setup a Web server for a school project. They’re teaching pupils HTML and CSS and she wanted each of them to have their own space to test what they’ve done. There was one problem: she doesn’t know anything about system administration. So I had to setup something where she would not touch the configuration or restart Nginx.

The idea was simple: I wanted to add as many sites as I wanted without adding a configuration file or restarting Nginx.

Challenge accepted.

Things going on, I was thinking further and further, testing more and more things to know where I could go.

Dynamic vhosts for static sites Dynamic vhosts for Ruby on Rails apps with Passenger Dynamic vhosts with multiple backends Dynamic vhosts with SSL support Dynamic vhosts with their own error pages (and fallback) Dynamic vhosts with separated logging Dynamic vhosts with basic auth

I setup different configurations and found out some things were possible and some other were not because of how Nginx handles some .

1. The good: dynamic vhosts for static sites

The following configuration file is a very basic setup for static Web sites. There’s no backend here, no SSL, nothing fancy, the only goal was to check the setup works (and it does).

server {

listen 80;

server_name _; set $site_root /data/www/$host;



location / {

root $site_root;

}



error_log /var/log/nginx/error.log info;

access_log /var/log/nginx/access.log; }

server_name _ The `_` server_name is a _catch all_, which means it will process every query. If you want to process subdomains of your domain name, you can use `*.platyp.us` instead of `_` $site_root The location where sites are hosted. Using the `$host` variable allows to set that configuration directive dynamically. So `foo.bar` is hosted in `/data/www/foo.bar/`.

Note that $site_root is a user defined variables we use to avoid repeating ourselves. $host is a Nginx variable.

2. The bad: dynamic vhosts for Ruby on Rails apps with Passenger

If you want to host Ruby on Rails Web sites, Phusion Passenger mod_rails is a nice and convenient solution. I’ve been using it since the very beggining to host Publify blogs and never had anything to say about it.

I’ve reused the static site configuration, adding the minimal Passenger setup for clarity.

server {

listen 80;

server_name _; set $site_root /data/www/$host/public;

set $log_root $site_root/log;



location / {

root $site_root;

passenger_enabled on;

}



error_log $log_root/error.log info;

access_log $log_root/access.log main; }

2 lines differ from the first example

$site_root _ Ruby on Rails `root` lies in the application `public` directory so this is where Nginx root goes. passenger_enabled on; Enables `mod_rails` for the current location

Bad news, it didn’t work.

Despite being called mod_rails, Passenger does not work like its PHP counterpart. It needs to know which sites it manages at startup time to launch a Ruby worker for each of them.

3. The ugly: dynamic vhosts with multiple backends

So far, we’ve been playing with site hosted locally. This time, we’ll configure Nginx as a reverse proxy to access various application servers relying on Lua module and Redis as a backend. I’ve found the Lua configuration on Dan Sosedoff blog

So you’ll need:

Lua.

Nginx compiled with Lua support.

A Redis server running somewhere. If you only have one machine running your frontend server, localhost is definitely the place to be.

server {

listen 80;

server_name _;

set $log_root $site_root/logs;



location / {



root $site_root;



if (! -d $site_root) {

set $backend "";

rewrite_by_lua '

-- load global route cache into current request scope

-- by default vars are not shared between requests

local routes = _G.routes



if routes == nil then

routes = {}

ngx.log(ngx.ALERT, "Route cache is empty.")

end



local route = routes[ngx.var.http_host]

if route == nil then

local redis = require "redis"

local client = redis.connect("localhost", 6379)

route = client:get(ngx.var.host)

end



-- fallback to redis for lookups

if route ~= nil then

ngx.var.upstream = route

routes[ngx.var.http_host] = route

_G.routes = routes

else

ngx.exit(ngx.HTTP_NOT_FOUND)

end

';



proxy_buffering off;

proxy_redirect off;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_pass



set $log_root "/var/log/nginx";

}

}



error_log $log_root/error.log info;

access_log $log_root/access.log main; set $site_root /data/www/$host;set $log_root $site_root/logs;location / {root $site_root;if (! -d $site_root) {set $backend "";rewrite_by_lua '-- load global route cache into current request scope-- by default vars are not shared between requestslocal routes = _G.routesif routes == nil thenroutes = {}ngx.log(ngx.ALERT, "Route cache is empty.")endlocal route = routes[ngx.var.http_host]if route == nil thenlocal redis = require "redis"local client = redis.connect("localhost", 6379)route = client:get(ngx.var.host)end-- fallback to redis for lookupsif route ~= nil thenngx.var.upstream = routeroutes[ngx.var.http_host] = route_G.routes = routeselsengx.exit(ngx.HTTP_NOT_FOUND)end';proxy_buffering off;proxy_redirect off;proxy_set_header X-Real-IP $remote_addr;proxy_set_header Host $host;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_pass http://$backend; set $log_root "/var/log/nginx";error_log $log_root/error.log info;access_log $log_root/access.log main; }

This one is more complicated because of the Lua part.

if (! -d $site_root) If the site is not a static one (or a Passenger powered Ruby on Rails site), there’s no reason the directory exists, so we know we’ll have to proxy. set $backend “” Sets an empty var for the backend. This var is later set by the Lua script using the cache fetched from Redis. If no backend is available, then we return a 404. proxy_pass http://$backend Pass the query to the needed proxy. set $log_root “/var/log/nginx” This is a fallback to store the logs into `/var/log/nginx` if `/data/www/$host` does not exist.

Now let’s test:

$ ./redis-cli

redis> set foo.platyp.us 192.168.0.1

OK

redis> set bar.platyp.us 192.168.0.2

OK $ curl foo.platyp.us

Hits 192.168.0.1

$ curl bar.platyp.us

Hits 192.168.0.1

$ curl perry.platyp.us

Error 404

I’m not a great fan of this solution so I call it ugly.

First, it breaks my catch all philosophy as you need to map every site with a Redis record. It means no more handy wildcard.

Second, I’m not fan of the if (! -d $site_root) which makes use of Nginx rewrite module. Nginx uses stat(2) which checks the nature of the file against the Posix macro S_ISDIR so we’re adding additional system calls for every http request. It’s not a big deal, but it’s better to know how it works.

The good news is you can handle non existing sites on Nginx level as you only use proxy_pass if and only if:

There’s no static Web site with that name. There’s a backend to handle it.

Nginx documentation states that if is evil and I should use try_files instead. Unfortunately, try_files won’t work the way I want.

My first tests relied on a DNS based configuration. Every Web site had a $host.internal URI pointing to the right backend. I didn’t like it for at least 3 reasons:

DNS adds lots of complexity as you need to create a new A (or AAAA) record for every Web site you host to know where it should forward the requests, and you don’t always control it. Most IT will refuse to dynamically add DNS entries every time you add a new site. Adding lines in /etc/hosts is not a solution either. DNS adds network latency. The Nginx LUA module allows to caches the results so you don’t have to query Redis every time you need to map a Web site with its backend.

4. The bad: dynamic vhosts with SSL support

Thanks to TLS SNI ((RFC 6066)[https://tools.ietf.org/html/rfc6066]), you can now manage multiple certificates on the same IP address. SNI does not work with old browsers, but it’s a great alternative the IPv4 shortage if you don’t care about the minority still using Internet Explorer 6.

In a dynamic SSL or not scope, the most obvious configuration was:

server {

listen 80;

server_name _;

set $ssl_root $site_root/ssl;



if (-f $ssl_root/$host.pem) {

return 301

}



location / {

root $site_root;

}



error_log /var/log/nginx/error.log info;

access_log /var/log/nginx/access.log; set $site_root /data/www/$host;set $ssl_root $site_root/ssl;if (-f $ssl_root/$host.pem) {return 301 https://$host$request_uri; location / {root $site_root;error_log /var/log/nginx/error.log info;access_log /var/log/nginx/access.log; } server {

listen 443;

server_name _; set $site_root /data/www/$host;

set $ssl_root $site_root/ssl;



ssl on;

ssl_certificate $ssl_root/$host.pem;

ssl_certificate_key $ssl_root/$host.key;



location / {

root $site_root;

}



error_log /var/log/nginx/error.log info;

access_log /var/log/nginx/access.log; }

The SSL configuration is the minimal one. If you’re looking for a more complete one, I’ve written A Bulletproof Nginx SSL Configuration you can use.

Unfortunately, dynamic vhosts with SSL won’t work. Trying to start Nginx with this configuration fails with:

nginx: [emerg] BIO_new_file("$ssl_root/$host.pem;") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('$ssl_root/$host.pem;','r') error:2006D080:BIO routines:BIO_new_file:no such file)

nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed

There are 2 reasons for this:

Nginx needs to load the whole SSL server configuration at start time, so it throws an error when the certificate or key does not exist. The Nginx SSL configuration parser does not expand user defined variables so it needs a relative or absolute path.

5. The good: dynamic vhosts with their own error pages (and fallback)

Nginx default configuration provides a handy way to manage custom error pages in http, server, location and if in location contexts. The following example is designed for a custom 404 page but it can be easily extended to any 40x or 50x pages.

server {

listen 80;

server_name _; set $site_root /data/www/$host;



location / {

root $site_root;

}



error_page 404 =404 /404.html;



location /404.html {

root $site_root/error_files;

internal;



error_page 404 =404 @fallback_404;

}



location @fallback_404 {

root /var/www/;

try_files /404.html =404;

internal;

}



error_log /var/log/nginx/error.log info;

access_log /var/log/nginx/access.log; }

error_page 404 =404 /404.html; Tells Nginx to use `/404.html` in case of `HTTP_NOT_FOUND` with a 404 return code. location /404.html What happens when hitting /404.html. This is where the fun beggins. root $site_root/error_files; Changes the location `root` to match the Web site `error_pages` directory. internal; Means it’s an internal redirection so the redirect is invisible client side. error_page 404 =404 @fallback_404; In the `/404.html` location, the error page is in the named location `@fallback_404` and returns a 404 http code. location @fallback_404 This is the named location used to configure the fallback 404 page. In this location, the `root` is changed to `/var/www/` so it will read files from that path instead of `$site_root` try_files /404.html =404; Returns `/var/www/404.html` if it exists with a 404 http code.

The most obscure part is internal. According to Nginx documentation :

Specifies that a given location can only be used for internal requests. For external requests, the client error 404 (Not Found) is returned. Internal requests are the following:

requests redirected by the error_page, index, random_index, and try_files directives;

requests redirected by the “X-Accel-Redirect” response header field from an upstream server;

subrequests formed by the “include virtual” command of the ngx_http_ssi_module module and by the ngx_http_addition_module module directives;

requests changed by the rewrite directive.

And also:

There is a limit of 10 internal redirects per request to prevent request processing cycles that can occur in incorrect configurations. If this limit is reached, the error 500 (Internal Server Error) is returned. In such cases, the “rewrite or internal redirection cycle” message can be seen in the error log.

6. The bad: dynamic vhosts with separated logging

Last thing I’ve tried to do was to dynamically separate the logs by server. I thought it would be interesting to let the users access their logs for debugging or processing purpose.

Let’s improve the first configuration.

server {

listen 80;

server_name _; set $site_root /data/www/$host;

set $logging_root $site_root/logs;



location / {

root $site_root;

}



error_log $logging_root/error.log info;

access_log $logging_root/access.log; }

If this is your only server, Nginx won’t start. It won’t start because it needs to open the log files at startup, and the access_log and error_log options don’t expand variables. There’s a solution though, which is delegating the log processing to rsyslog or syslog-ng but that’s beyond what I wanted to talk about here.

7. The bad: dynamic vhosts with basic auth

server {

listen 80;

server_name _; set $site_root /data/www/$host;



location / {

root $site_root;



if (-f $site_root/.htpasswd) {

auth_basic "Restricted";

auth_basic_user_file $site_root/.htpasswd;

}

}



error_log /var/log/nginx/error.log info;

access_log /var/log/nginx/access.log; }

Forget about it, it won’t work.

$ nginx -t

nginx: [emerg] "auth_basic" directive is not allowed here in /usr/local/etc/nginx/nginx.conf:124

nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed

The reason why it fails is because if is not part of the general configuration module as one should believe. if is part of the rewrite module and auth_basic is another module. That’s one of the reason why the Nginx community thinks if is evil.

I guess I’m done, or almost.

If I’ve been missing something or there’s way to do one of the things I’ve failed at with my “never configure, never restart” philosophy, please drop me an email frederic@t37.net, I’ll be happy to update the article.

There’s one more thing I’ve been playing with part of the week-end, and I hope it will stay buried from the man knowledge until the end of time. Any real work implementation of what I’ve been testing would certainly result in opening a door to our world for the Great Old Ones. I’ve started to implement Apache .htaccess to Nginx using Lua.

Original article published on The good, the bad and the ugly of virtual hosting with Nginx