[Kepler-Project] Re: Basic benchmark result

Guillame; Herewith my notes. You can instruct Lighttpd to launch as many FCGI workers as you like, but it's a little counter-intuitive. Specify it with the "max-procs" setting in the fastcgi.server. I was interested by your results, and so decided to have a little shootout of my own. Here are my results for plain HTML, PHP and Lua. ---------------------------------------------------------------------- Plain HTML, default Lighttpd on Ubuntu VM: ab -c 10 -n 10000 http://10.1.1.6/hello_world.html ... Requests per second: 3163.00 [#/sec] (mean) Time per request: 3.162 [ms] (mean) Time per request: 0.316 [ms] (mean, across all concurrent requests) Transfer rate: 1000.79 [Kbytes/sec] received ---------------------------------------------------------------------- Plain PHP with default Lighttpd settings: ab -c 10 -n 10000 http://10.1.1.6/hello_world.php ... Requests per second: 1431.15 [#/sec] (mean) Time per request: 6.987 [ms] (mean) Time per request: 0.699 [ms] (mean, across all concurrent requests) Transfer rate: 373.24 [Kbytes/sec] received ---------------------------------------------------------------------- Plain PHP, with PHP_FCGI_CHILDREN increased to 16: ab -c 10 -n 10000 http://10.1.1.6/hello_world.php ... Requests per second: 1388.17 [#/sec] (mean) Time per request: 7.204 [ms] (mean) Time per request: 0.720 [ms] (mean, across all concurrent requests) Transfer rate: 361.95 [Kbytes/sec] received ---------------------------------------------------------------------- PHP, default settings, with XCache opcode cache installed: ab -c 10 -n 10000 http://10.1.1.6/hello_world.php ... Requests per second: 1076.88 [#/sec] (mean) Time per request: 9.286 [ms] (mean) Time per request: 0.929 [ms] (mean, across all concurrent requests) Transfer rate: 280.81 [Kbytes/sec] received ---------------------------------------------------------------------- Now the interesting stuff -- using Lua. This involved a lot of futzing about with ubuntu. 9.04 intrepid includes promising-looking packages for WSAPI, but they are incomplete. Apparently 9.10 karmic will have integrated WSAPI packages which get over the "it just works" hurdle that I am able to vault by myself. My installation is thus luarocks-based. As has been noted on this mailing list (thanks Google!), this means installing luafilesystem and rings, both of which are required, neither of which are automatic dependencies. I consider this a bug. Anyhow, moving along. ---------------------------------------------------------------------- First I run the microbenchmark with the default 4 wsapi.fcgi instances: ab -c 10 -n 10000 http://10.1.1.6/hello_world.ws ... Requests per second: 1664.60 [#/sec] (mean) Time per request: 6.007 [ms] (mean) Time per request: 0.601 [ms] (mean, across all concurrent requests) Transfer rate: 374.03 [Kbytes/sec] received ---------------------------------------------------------------------- And now at max-procs = 16: ab -c 10 -n 10000 http://10.1.1.6/hello_world.ws ... Requests per second: 1526.44 [#/sec] (mean) Time per request: 6.551 [ms] (mean) Time per request: 0.655 [ms] (mean, across all concurrent requests) Transfer rate: 342.92 [Kbytes/sec] received ---------------------------------------------------------------------- And now at max-procs = 32: ab -c 10 -n 10000 http://10.1.1.6/hello_world.ws ... Requests per second: 1487.43 [#/sec] (mean) Time per request: 6.723 [ms] (mean) Time per request: 0.672 [ms] (mean, across all concurrent requests) Transfer rate: 334.26 [Kbytes/sec] received In this latter test I looked at load. It never breaks past 0.07. ---------------------------------------------------------------------- Now max-procs = 32 with higher concurrency: ab -c 50 -n 10000 http://10.1.1.6/hello_world.ws Requests per second: 1434.58 [#/sec] (mean) Time per request: 34.854 [ms] (mean) Time per request: 0.697 [ms] (mean, across all concurrent requests) Transfer rate: 322.32 [Kbytes/sec] received Not a huge improvement, but then, not much degradation either. ---------------------------------------------------------------------- Last of all, something a bit different. The lighttpd "mod_magnet" module allows you to embed lua scripts directly into the web server. Let's see how that performs: ab -c 10 -n 10000 http://10.1.1.6/hello_world ... Requests per second: 3005.95 [#/sec] (mean) Time per request: 3.327 [ms] (mean) Time per request: 0.333 [ms] (mean, across all concurrent requests) Transfer rate: 675.37 [Kbytes/sec] received Notice that this is twice as fast as any other FastCGI configuration and only slightly slower than raw HTML. ---------------------------------------------------------------------- The same again, with much higher concurrency: ab -c 250 -n 10000 http://10.1.1.6/hello_world ... Requests per second: 2653.38 [#/sec] (mean) Time per request: 94.220 [ms] (mean) Time per request: 0.377 [ms] (mean, across all concurrent requests) Transfer rate: 596.03 [Kbytes/sec] received Still much better than FCGI. There is a big "gotcha" for this, however. Lighttpd is single-threaded by design, and the pass through a lua script is a single-threaded pass. In scenarios where you have a long-running lua script, this will be worse than the fcgi scenario. ---------------------------------------------------------------------- The busy-wait script, run on FCGI with 16 wsapi instances: ab -c 10 -n 10000 http://10.1.1.6/busy_world.ws ... Requests per second: 954.08 [#/sec] (mean) Time per request: 10.481 [ms] (mean) Time per request: 1.048 [ms] (mean, across all concurrent requests) Transfer rate: 214.32 [Kbytes/sec] received ---------------------------------------------------------------------- The busy-wait script, run inside lighttpd with mod_magnet: ab -c 10 -n 1000 http://10.1.1.6/busy_world Requests per second: 5.20 [#/sec] (mean) Time per request: 1922.162 [ms] (mean) Time per request: 192.216 [ms] (mean, across all concurrent requests) Transfer rate: 1.17 [Kbytes/sec] received The results are catastrophically worse for this case -- it was so slow that I killed the original test and ran it again with only 1000 requests (10x less). This is because every request is dealt with by itself. The normal model allows lighty to push a job onto a fastcgi worker, let them run while it services the next HTTP request, then send the reply when it's ready. mod_magnet breaks this model. You have been warned. ---------------------------------------------------------------------- My conclusions: * Lua is faster than PHP in a FastCGI environment * Lua probably scales better -- compare the multi-worker examples. * Embedding lua in lighttpd using mod_magnet can be incredibly fast ... * ... but does not scale for long-running scripts. * I should be studying for my exams, instead of spending a morning on this stuff. Cheers, JC