Prerequisites

Used tools

nodejs

Varnish cache

Monit

upstart

Notes

Setting up the environment

Nodejs

$ sudo apt-get install python g++ make $ wget http://nodejs.org/dist/node-latest.tar.gz $ tar xvfvz node-latest.tar.gz $ cd node-v0.xx.xx (replace a version with your own) $ ./configure $ make $ sudo make install 1 2 3 4 5 6 7 $ sudo apt - get install python g ++ make $ wget http : / / nodejs .org / dist / node - latest .tar .gz $ tar xvfvz node - latest .tar .gz $ cd node - v0 .xx .xx ( replace a version with your own ) $ . / configure $ make $ sudo make install

Monit

$ apt-get install monit 1 $ apt - get install monit

Varnish cache

$ apt-get install apt-transport-https $ curl https://repo.varnish-cache.org/ubuntu/GPG-key.txt | apt-key add - $ echo "deb https://repo.varnish-cache.org/ubuntu/ precise varnish-4.0" >> /etc/apt/sources.list.d/varnish-cache.list $ apt-get update $ apt-get install varnish 1 2 3 4 5 $ apt - get install apt - transport - https $ curl https : / / repo .varnish - cache .org / ubuntu / GPG - key .txt | apt - key add - $ echo "deb https://repo.varnish-cache.org/ubuntu/ precise varnish-4.0" >> / etc / apt / sources .list .d / varnish - cache .list $ apt - get update $ apt - get install varnish

$ apt-get install python-pip python-dev build-essential $ pip install --upgrade pip $ pip install --upgrade virtualenv $ pip install docopt==0.6.1 1 2 3 4 $ apt - get install python - pip python - dev build - essential $ pip install -- upgrade pip $ pip install -- upgrade virtualenv $ pip install docopt == 0.6.1

Setting up the platform

I use a sub domain for each application,it also make migration easy (here we use helloapp.nodeserver.com) each application resides in /opt/nodeapps/<subdomain name>/, (here we'll use /opt/nodeapps/helloapp.nodeserver.com when a service require special configuration file for the application, I use the subdomain name as configuration file name (see bellow the upstart job and monit configurations) when a variable designate my application in a script I use something like mysubdomain_mydomain_com

"hello world" server

/opt/nodeapps/hello.server/hello.js var http = require('http'); var server = http.createServer(function (request, response) { console.log("received request ", request); response.writeHead(200, {"Content-Type": "text/plain"}); response.end("Hello World

"); }); // Listen on port 10000 server.listen(10000); console.log("Server running at http://127.0.0.1:10000/"); 1 2 3 4 5 6 7 8 9 10 11 12 13 14 var http = require ( 'http' ) ; var server = http . createServer ( function ( request , response ) { console . log ( "received request " , request ) ; response . writeHead ( 200 , { "Content-Type" : "text/plain" } ) ; response . end ( "Hello World

" ) ; } ) ; // Listen on port 10000 server . listen ( 10000 ) ; console . log ( "Server running at http://127.0.0.1:10000/" ) ;

Creating upstart job

/etc/init/my.nodeserver.com.conf #!upstart description "node.js Hello Server" author "Ezelia" start on startup stop on shutdown env APPNAME="helloapp.nodeserver.com" script export HOME="/opt/nodeapps/$APPNAME" echo $$ > /var/run/$APPNAME.pid /bin/bash <<EOT exec sudo -u nodeuser /usr/local/bin/node /opt/nodeapps/$APPNAME/server.js | /nodeapps/scripts/log.py /var/log/nodeapps/$APPNAME.log EOT end script pre-start script echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" | /nodeapps/scripts/log.py /var/log/nodeapps/$APPNAME.log end script pre-stop script rm /var/run/$APPNAME.pid echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" | /nodeapps/scripts/log.py /var/log/nodeapps/$APPNAME.log end script 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 #!upstart description "node.js Hello Server" author "Ezelia" start on startup stop on shutdown env APPNAME = "helloapp.nodeserver.com" script export HOME = "/opt/nodeapps/$APPNAME" echo $ $ > / var / run / $APPNAME .pid / bin / bash << EOT exec sudo - u nodeuser / usr / local / bin / node / opt / nodeapps / $APPNAME / server .js | / nodeapps / scripts / log .py / var / log / nodeapps / $APPNAME .log EOT end script pre - start script echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" | / nodeapps / scripts / log .py / var / log / nodeapps / $APPNAME .log end script pre - stop script rm / var / run / $APPNAME .pid echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" | / nodeapps / scripts / log .py / var / log / nodeapps / $APPNAME .log end script

you can use this same upstart script to create multiple applications upstart jobs, if you respect the same structure as mine, all you have to do is to change APPNAME variable

$ start helloapp.nodeserver.com 1 $ start helloapp .nodeserver .com

Configuring varnish cache

DAEMON_OPTS="-a :6081 \ 1 DAEMON_OPTS = " - a : 6081 \

DAEMON_OPTS="-a :80 \ 1 DAEMON_OPTS = " - a : 80 \

/etc/varnish/default.vcl /* section 1 */ backend helloapp_nodeserver_com { .host = "127.0.0.1"; .port = "10000"; } /************/ sub vcl_fetch { set beresp.ttl = 8h; set beresp.grace = 600s; if (beresp.status == 404 || beresp.status >= 500 || beresp.status == 503) { set beresp.ttl = 0s; } return (deliver); } sub vcl_deliver { /* remove those unneeded tags for production */ remove resp.http.X-Varnish; remove resp.http.Via; remove resp.http.Age; remove resp.http.X-Powered-By; return (deliver); } sub vcl_recv { if (req.url == "/get-server-health") { error 200 "Server UP"; } /* section 2 */ if (req.http.host == "helloapp.nodeserver.com") { set req.backend = helloapp_nodeserver_com; } /*************/ if (req.http.Upgrade ~ "(?i)websocket") { unset req.http.Cookie; return (pipe); } if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Authorization || req.http.Cookie) { /* Not cacheable by default */ return (pass); } return (lookup); } sub vcl_pipe { if (req.http.upgrade) { set bereq.http.upgrade = req.http.upgrade; } } sub vcl_pass { return (pass); } sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; #if (obj.status == 200) return (deliver); set obj.http.Retry-After = "5"; synthetic {" <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html> <head> <title>Server Offline</title> </head> <body> <h2 style="display:block;text-align:center;border:2px solid #c22;color:#911;background:#ff7777">This application is offline :(</h2> <pre> "} + obj.status + " " + obj.response + {" </pre> <hr /> </body> </html> "}; return (deliver); } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 /* section 1 */ backend helloapp_nodeserver_com { . host = "127.0.0.1" ; . port = "10000" ; } /************/ sub vcl_fetch { set beresp . ttl = 8h ; set beresp . grace = 600s ; if ( beresp . status == 404 || beresp . status >= 500 || beresp . status == 503 ) { set beresp . ttl = 0s ; } return ( deliver ) ; } sub vcl_deliver { /* remove those unneeded tags for production */ remove resp . http . X - Varnish ; remove resp . http . Via ; remove resp . http . Age ; remove resp . http . X - Powered - By ; return ( deliver ) ; } sub vcl_recv { if ( req . url == "/get-server-health" ) { error 200 "Server UP" ; } /* section 2 */ if ( req . http . host == "helloapp.nodeserver.com" ) { set req . backend = helloapp_nodeserver_com ; } /*************/ if ( req . http . Upgrade ~ "(?i)websocket" ) { unset req . http . Cookie ; return ( pipe ) ; } if ( req . restarts == 0 ) { if ( req . http . x - forwarded - for ) { set req . http . X - Forwarded - For = req . http . X - Forwarded - For + ", " + client . ip ; } else { set req . http . X - Forwarded - For = client . ip ; } } if ( req . request != "GET" && req . request != "HEAD" && req . request != "PUT" && req . request != "POST" && req . request != "TRACE" && req . request != "OPTIONS" && req . request != "DELETE" ) { /* Non-RFC2616 or CONNECT which is weird. */ return ( pipe ) ; } if ( req . request != "GET" && req . request != "HEAD" ) { /* We only deal with GET and HEAD by default */ return ( pass ) ; } if ( req . http . Authorization || req . http . Cookie ) { /* Not cacheable by default */ return ( pass ) ; } return ( lookup ) ; } sub vcl_pipe { if ( req . http . upgrade ) { set bereq . http . upgrade = req . http . upgrade ; } } sub vcl_pass { return ( pass ) ; } sub vcl_error { set obj . http . Content - Type = "text/html; charset=utf-8" ; #if (obj.status == 200) return (deliver); set obj . http . Retry - After = "5" ; synthetic { " <? xml version = "1.0" encoding = "utf-8" ?> <!DOCTYPE html PUBLIC " - //W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd" > < html > < head > < title > Server Offline < / title > < / head > < body > < h2 style = "display:block;text-align:center;border:2px solid #c22;color:#911;background:#ff7777" > This application is offline : ( < / h2 > < pre > "} + obj.status + " " + obj.response + {" < / pre > < hr / > < / body > < / html > " } ; return ( deliver ) ; }

/etc/init.d/varnish start 1 / etc / init .d / varnish start

$ stop helloapp.nodeserver.com 1 $ stop helloapp .nodeserver .com

Monit to the rescue

/etc/monit/conf.d/hello.server.conf check process helloapp_nodeserver_com with pidfile "/var/run/helloapp.nodeserver.com.pid" start program = "/sbin/start helloapp.nodeserver.com" stop program = "/sbin/stop helloapp.nodeserver.com" if failed port 10000 protocol HTTP request / with timeout 30 seconds then restart 1 2 3 4 5 6 7 check process helloapp_nodeserver_com with pidfile "/var/run/helloapp.nodeserver.com.pid" start program = "/sbin/start helloapp.nodeserver.com" stop program = "/sbin/stop helloapp.nodeserver.com" if failed port 10000 protocol HTTP request / with timeout 30 seconds then restart

service monit start 1 service monit start

/etc/monit/conf.d/varnish.conf check process varnish with pidfile "/var/run/varnishd.pid" start program = "/etc/init.d/varnish start" stop program = "/etc/init.d/varnish stop" if failed host 127.0.0.1 port 80 protocol HTTP and request "/get-server-health" then restart 1 2 3 4 5 6 check process varnish with pidfile "/var/run/varnishd.pid" start program = "/etc/init.d/varnish start" stop program = "/etc/init.d/varnish stop" if failed host 127.0.0.1 port 80 protocol HTTP and request "/get-server-health" then restart

service monit restart 1 service monit restart

Final step : configur log rotation

/var/log/nodeapps/*.log { daily create rotate 30 compress size 10M missingok notifempty sharedscripts } 1 2 3 4 5 6 7 8 9 10 / var / log / nodeapps / * . log { daily create rotate 30 compress size 10M missingok notifempty sharedscripts }

Live example

Multiple node apps on the same server

Create the upstart job Configure varnish (declare a new backend and define the request hostname to use) Create a monit configuration file to monitor the application

Your turn !

If you are reading this you already know that nodejs is a powerful engine for your realtime application or multiplayer game. But it's not easy to setup an up and running platform, which is self monitored. by self monitored here I mean that the platform will automatically detect a nodejs crash/stop and will restart the service. The platform should also support load, and for this we'll use Varnish to optimize static content delivery.This tutorial suppose that you are already familiar with linux (I'll use Ubuntu here) : know how to install/remove package from package manager and sources, and how to edit configuration files. obviously, some programming and networking skills are preferable, but not required.I'll suppose that you already own a dedicated server/VM for your node application That. that helloapp.nodeserver.com point to the server ip And that you want your node application to run on helloapp.nodeserver.com Let's startthe first step is to install the tools we'll need to build our platformnodejs can be installed using the package manager (apt-get, yum ...etc) but I'd recommande to install it from sources to get the latest version. here is how you setup nodejs from sources :monit is a little daemon which monitor services and restart them when a configured condition is met. to setup monitVarnish is an HTTP cache proxy, it'll accelerate the delivery of static content by caching them while reducing nodejs server load. installing vernish is straightforward, here are the instruction from official web site :We'll also need some python mainly to be able to correctly log our nodejs server activityall needed tools are now installed, let's start the server configurationBefore starting, it's very important to think of a structured organization of your platform, this will make teamwork easier because most of the time, you'll not be alone on projects requiring such platforms. event if you are alone, be sure that some months later, you'll forget how you made things if they are not well structured and documented ... so here is the structure I use, you don't have to follow the exact same structure, It's here to give you an idea:the rest of this article will illustrate how I follow this structure. now we can start the configuration.For testing purpose, I'll create a minimal nodejs http server which represent your real-time nodejs app or you multiplayer game server. I recommand to put all nodejs apps in a separate directory. here we'll useour hello world http server create file /opt/nodeapps/helloapp.nodeserver.com/server.jsnow we'll write the upstart job that'll be responsible of launching our helloapp server. this will allow you if you want, to schedule the nodejs startup in a system runlevel as any other system service (but it's not needed since we'll use monit), the upstart will also send helloapp server output to a logfile so we can inspect what's hapening, investigate crashes ...etc for logging, I'm using a little python script that you can download here. this script will pipe the logs to logfiles and ensure that they are not locked. you can just pipe stdout/strerr directly to logfiles but then you'll have some troubles (and headache) if you want to make rotating logs. bellow I suppose that the script is in /opt/nodeapps/scripts/log.py create/edit file /etc/init/helloapp.nodeserver.com.conftesting the upstart jobgo to your browser and type : http://helloapp.nodeserver.com:10000/ note at this step, the server is accessible throught 10000 port, but this will change with Varnish Cache.by default, varnish run on port 6081 if you want your application to be accessible from default HTTP port (or some other port different from 6081) you have to edit /etc/default/varnish and changetofor port 80 next we need to configure varnish this is a little tricky here is my varnish config which is compatible with nodejs, and handle websockets correctly (since websockets are mandatory for the application/game server we are making 😉 )the "section 1" declare a backend, the "section 2" identifies requests hostname which will be handled by our backend. now start varnishand go to http://helloapp.nodeserver.com/ (if you used a different port than 80 you have to specify it) your hello server should respond. we are almost done ! now what if the server crash or stop ? let's simulate a server stopnow when you go to helloapp.nodeserver.com the you get a varnish error page (btw, you can use it to show a message to your users 😉 )global monit configuration create/edit file /etc/monit/conf.d/helloapp.nodeserver.com.confwhat this script do is to tell monit to check the presence of helloapp.nodeserver.com process that is listening to local port 10000, if no response within 30 seconds restart the server. save the file and start monitnow go to to http://helloapp.nodeserver.com/ and you'll see that the server is responding. try to stop it : stop helloapp.nodeserver.com refresh the page .... and he's here again 🙂 but wait, our nodejs application depend on both nodejs server AND varnish cash let's monitor varnish create/edit file /etc/monit/conf.d/varnish.confhere the check condition is different, we actually check varnish presence using a special request we configured in varnish config. restart monitnow if everything is up and running, just make sure that monit is starting on system boot : update-rc.d monit enablelog are very important in such applications, and linux provide an efficient tool to handle log rotations to prevent infinite growing log files. to configure log rotation for all our nodejs applications create/edit /etc/logrotate.d/nodeappswant to see a live example ? here is doyazan prototype : http://demo.ezelia.com/ which is a proof of concept for a multiplayer HTML5 isometric game 🙂 open the url, enter a random login and click start under the desirez avatar.Now that you have a working platform, you can easily add other nodejs applications. for this you have three steps :You can automate those operations using a python or perl script, but beware, you should make sure that your script are properly tested, a misconfiguration in varnish can make all your node apps unreachable.this article explained the basics to obtain a self monitored nodejs server for your application. one important thing that has not been adressed here is security, since it's a BIG topic... but you can add some security to your current server with a little more work. First, a good practive would be to drop all incomming connexions to port different from 80, since all your node applications should be accessible throught this port now. For hack attempts, or bad behaviours, what I usually do, is to let the nodejs application log every suspect activity. those activities are tagged with some keywords (you can prefix suspect activities log with somthing like [SECURITY-WARN]) then I use fail2ban with a custom rule to detect and ban IPs generating a lot of those security warnings this can be a good exercice to enhance the platform 🙂