Optimize VPS server for Drupal Hosting | Using Varnish Written by Guillermo Garron

Date: 2012-04-08 14:26:00 00:00

Introduction

In the past, and for some years I've run my blog with the help of Drupal, in that time and in the first months, I've got Slashdoted, and Dugg three times, all three times my server went down.

Since Then I've become obsessed with tweaking my server configuration to support the load of Slashdot, Digg and the like.

I'm not running my blog over Drupal anymore, but I still like Drupal a lot, and this weekend I've been playing with Drupal 7 and Varnish, to see how it performs, under heavy load.

I was trying to figure out a way to optimize the Drupal configuration without having to tweak too much into the Drupal or server configuration, and without the need to add too many "performance" modules.

The environment

Here is my configuration details:

Arch Linux 2011.10

RackSpace VPS

256 RAM

Apache/PHP/MySQL/Varnish

Configuration

Drupal

I'm using the basic Drupal 7 installation, with core cache turned ON.

LAMP

LAMP is the standard available in Arch Linux by the time of this writing, and no special configuration to any of the components. Except that Apache is listening to port 8080 instead of port 80. So it can server pages internally to Varnish.

Varnish

From Wikipedia:

Varnish is an HTTP accelerator designed for content-heavy dynamic web sites. In contrast to other HTTP accelerators, such as Squid, which began life as a client-side cache, or Apache and nginx, which are primarily origin servers, Varnish was designed from the ground up as an HTTP accelerator. Varnish is focused exclusively on HTTP, unlike other proxy servers that often support FTP, SMTP and other network protocols

Varnish is going to support the load, but once again the configuration is pretty basic:

/etc/conf.d/varnish listing:

VARNISHD_OPTS="-a 0.0.0.0:80 \ -b localhost:8080 \ -T localhost:6082 \ -s malloc,64M \ -s file,/var/cache/$INSTANCE/varnish_storage.bin,1G \ -u nobody -g nobody" VARNISH_CFG="/etc/varnish/default.vcl"

/etc/varnish/default.vcl listing:

backend default { .host = "127.0.0.1"; .port = "8080"; } sub vcl_recv { // Remove has_js and Google Analytics __* cookies. set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+| has_js)=[^;]*", ""); // Remove a ";" prefix, if present. set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); // Remove empty cookies. if (req.http.Cookie ~ "^\s*$") { unset req.http.Cookie; } // Cache all requests by default, overriding the // standard Varnish behavior. // if (req.request == "GET" || req.request == "HEAD") { // return (lookup); // } } sub vcl_hash { if (req.http.Cookie) { set req.hash += req.http.Cookie; } }

Testing

I've used ab tool to test, as this is a test to prove the Drupal site will be able to manage a spike in traffic from Digg or John Grubber, then ab is OK. If you plan to have thousands of pages and ten thousands pages views per hour, distributed all across the content, this may not be for you, but if only one or a few pages are popular at a time, this is the right place to be.

This is the command:

ab -n 10000 -c 100 http://10.179.138.178/drupal/?q=node/2

Where:

-n: Number of requests -c: Number of concurrent sessions

The results:

Server Software: Apache/2.2.22 Server Hostname: 10.179.138.x Server Port: 80 Document Path: /drupal/?q=node/2 Document Length: 8866 bytes Concurrency Level: 100 Time taken for tests: 39.358 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 95050000 bytes HTML transferred: 88660000 bytes Requests per second: 254.08 [#/sec] (mean) Time per request: 393.577 [ms] (mean) Time per request: 3.936 [ms] (mean, across all concurrent requests) Transfer rate: 2358.43 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 1 115 205.8 101 4696 Processing: 3 277 139.6 297 2946 Waiting: 1 141 129.8 147 2546 Total: 6 392 242.9 399 4948 Percentage of the requests served within a certain time (ms) 50% 399 66% 400 75% 400 80% 401 90% 448 95% 450 98% 499 99% 1399 100% 4948 (longest request)

After this, I've transfer that same page to an Nginx server running on a mirror Arch Linux powered server.

I've done that using curl

curl 10.179.138.178/drupal/?q=node/2 > /srv/http/drupal.html

And then run ab against Nginx with the static page, the result was:

Server Software: nginx/1.0.14 Server Hostname: 10.179.129.x Server Port: 8080 Document Path: /drupal.html Document Length: 8866 bytes Concurrency Level: 100 Time taken for tests: 38.353 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 91860000 bytes HTML transferred: 88660000 bytes Requests per second: 260.73 [#/sec] (mean) Time per request: 383.533 [ms] (mean) Time per request: 3.835 [ms] (mean, across all concurrent requests) Transfer rate: 2338.97 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 1 129 332.9 101 9051 Processing: 1 253 80.0 249 749 Waiting: 1 130 63.7 141 617 Total: 4 382 340.4 350 9300 Percentage of the requests served within a certain time (ms) 50% 350 66% 399 75% 400 80% 400 90% 401 95% 450 98% 652 99% 901 100% 9300 (longest request)

As you can see even though Drupal is not using boost, and it is full dynamic content, Varnish is making it equivalent to a static site. The results are almost the same in both tests.

Just to let you see how it performs without Varnish, here is what happens when Varnish is taken aside and Apache/PHP/MySQL support the full load.

Well: with the same load, MySQL hanged up, and all Operating System halted. I had to reboot the server from the Console.

So lowering the load:

ab -n 1000 -c 20 http://10.179.138.178/drupal/?q=node/2

The result is:

Server Software: Apache/2.2.22 Server Hostname: 10.179.138.178 Server Port: 80 Document Path: /drupal/?q=node/2 Document Length: 8866 bytes Concurrency Level: 20 Time taken for tests: 13.022 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 9445000 bytes HTML transferred: 8866000 bytes Requests per second: 76.79 [#/sec] (mean) Time per request: 260.443 [ms] (mean) Time per request: 13.022 [ms] (mean, across all concurrent requests) Transfer rate: 708.30 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 1 0.9 1 23 Processing: 32 259 757.4 109 6489 Waiting: 28 243 732.0 101 6305 Total: 34 260 757.4 110 6490 Percentage of the requests served within a certain time (ms) 50% 110 66% 120 75% 127 80% 132 90% 162 95% 1042 98% 3192 99% 4713 100% 6490 (longest request)

Conclusion###

As you can see, it is just a matter of install varnish with a very simple and basic configuration to improve server performance a lot. Being able to handle 250+ request per second in a 256 MB RAM server with Drupal CMS is not that difficult.

Once again, this is only valid for anonymous users, that is if you have a blog or a news or tutorial site, where your visitors does not need to be logged in to interact with your content. If you need this level of performance for logged users, then you need to look at memcached, APC and the like.

Note: All tests were run from another dedicated Cloud Server using internal IPs to access the Nginx and Apache servers, so there is no Bandwidth Limitation.