With an excellent foundation in place to automate provisioning of secure servers with Chef, I spent week two establishing an application project structure using the Node.js platform. Additionally, I setup a load balancer configuration with HAProxy and tested it using a local multi-machine Vagrant environment. Read on to find out details about this setup.

Value & Exit Criteria

As I mentioned in my previous post, I start each week defining the business value for the task at hand coupled with concise exit criteria to know when I’m done. This week’s task will ensure the conversion rate for subscribers is not impacted as the product usage grows. I consider the first impression of the product’s experience extremely important and I don’t want to see that disrupted as I try to scale a system in production.

To complete the task, four critera had to be met:

Setup a project structure for Node.js supporting a rapid development workflow and optimized production deployment.

Implement a Chef recipe to provision the Node.js application.

Implement a Chef recipe to provision HAProxy configured to load balance multiple Node.js application servers.

Test the Chef recipes using a local multi-machine Vagrant environment.

Setup a Node.js Project

Given my experience with Node.js from my previous job at Lockheed Martin, I selected it as the platform to build my product. I intend to implement a thin server architecture supporting a single-page application. Essentially, all the user interface logic will reside on the client using JavaScript libraries such as jQuery, require.js, and Backbone. The user interface will query an API layer built on the Express web application framework for Node.js.

Below is a step-by-step guide to setup a basic Node.js project using these libraries. It assumes that Node.js and npm are installed.

Setup Express. I began by creating a new directory, marinara, and initialized it as a git repository. Next, I created the package.json file which defines metadata for the project including its dependencies. Within that file I specified the Express version I required and then installed it using npm.

package.json 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 { "name" : "marinara" , "version" : "0.1.0" , "engines" : { "node" : "0.10.x" }, "dependencies" : { "express" : "~3.1.0" , "log" : "~1.3.1" , "underscore" : "~1.4.4" , "consolidate" : "~0.8.0" }, "devDependencies" : { } }

1 $ npm install

With Express installed, I created a server.js file at the root of the project directory and implemented a basic application with a simple logging and configuration strategy. Specifically, I defined configuration through environment variables (with suitable defaults) that will later be set with an Upstart script. I leveraged the log.js module to write messages using the log levels specified by syslog that will later be consolidated with an rsyslog server.

server.js 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 'use strict' ; var express = require ( 'express' ), Log = require ( 'log' ), api = require ( './lib/api' ), pkg = require ( './package.json' ), app = express (); // defines app settings with default values app . set ( 'log level' , process . env . MARINARA_LOG_LEVEL || Log . DEBUG ); app . set ( 'session secret' , process . env . MARINARA_SESSION_SECRET || 'secret' ); app . set ( 'session age' , process . env . MARINARA_SESSION_AGE || 3600 ); app . set ( 'port' , process . env . MARINARA_PORT || 8000 ); // configures default logger available for middleware and requests app . use ( function ( req , res , next ) { req . log = new Log ( app . get ( 'log level' )); next (); }); // logs all requests if log level is INFO or higher using log module format if ( app . get ( 'log level' ) >= Log . INFO ) { var format = '[:date] INFO :remote-addr - :method :url ' + ':status :res[content-length] - :response-time ms' ; express . logger . token ( 'date' , function () { return new Date (); }); app . use ( express . logger ( format )); } // TODO: setup cluster support app . listen ( app . get ( 'port' ));

Serve static assets and api requests. With the basic application structure complete, I implmented support to serve static assets both in development and production. During development, static assets are served from the app/ directory using require.js to keep code modular. For production, assets are optimized (concatnated and minified) and placed in the public/ directory. The single html page, index.html, is an underscore.js micro-template that replaces script and stylesheet references depending on the mode.

server.js 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 // configures underscore view engine // TODO: set caching for production app . engine ( 'html' , require ( 'consolidate' ). underscore ); app . set ( 'view engine' , 'html' ); app . set ( 'views' , __dirname + '/templates' ); // enables compression for all requests app . use ( express . compress ()); // serves index and static assets from app/ directory in development app . configure ( 'development' , function () { app . get ( '/' , function ( req , res ) { res . render ( 'index' , { jsFile : 'public/components/requirejs/require.js' , cssFile : 'public/styles/main.css' }); }); app . use ( '/public' , express . static ( __dirname + '/app' )); }); // serves index and static assets from optimized public/ directory in prod // TODO: replace staticCache middleware with varnish app . configure ( 'production' , function () { var oneYear = 60 * 60 * 24 * 365 , baseFile = 'public/' + pkg . name + '-' + pkg . version ; app . get ( '/' , function ( req , res ) { res . render ( 'index' , { jsFile : baseFile + '.min.js' , cssFile : baseFile + '.min.css' }); }); app . use ( '/public' , express . staticCache ()); app . use ( '/public' , express . static ( __dirname + '/public' , { maxAge : oneYear })); });

templates/index.html 1 2 3 4 5 6 7 8 9 10 11 <!DOCTYPE html> <html lang= "en" > <head> <meta charset= "UTF-8" > <title> Marinara </title> <link type= "text/css" rel= "stylesheet" href= "<%= cssFile %>" > </head> <body> <script data-main= "public/scripts/main" src= "<%= jsFile %>" ></script> </body> </html>

While I have not flushed out the api for the product yet, I went ahead and stubbed out a basic implementation. I created a sub-application in lib/api.js where I configured middleware to handle secure cookie sessions, cross-site request forgery, and body parsing. This sub-application is exported and mounted at the /api path on the main Express application in server.js.

lib/api.js 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 'use strict' ; var express = require ( 'express' ), app = express (); module . exports = function ( options ) { app . use ( express . cookieParser ()); app . use ( express . cookieSession ({ secret : options . sessionSecret , cookie : { maxAge : options . sessionAge } })); app . use ( express . bodyParser ()); app . use ( express . csrf ()); app . use ( app . router ); app . get ( '/hello' , function ( req , res ) { req . log . debug ( 'serving /api/hello request' ); res . send ( 'Hello Shawn' ); }); return app ; };

server.js 1 2 3 4 5 // mounts and configures rest api app . use ( '/api' , api ({ sessionSecret : app . get ( 'session secret' ), sessionAge : app . get ( 'session age' ) }));

Setup Bower and install client dependencies. With the server ready to handle requests, I shifted to the client-side by defining libraries in component.json that Bower, a browser package manager, would install. (I will discuss the installation of Bower shortly). I configured Bower (in the .bowerrc file) to install libraries to app/components so they could be served by Express during development.

component.json 1 2 3 4 5 6 7 8 9 10 11 { "name" : "marinara" , "version" : "0.1.0" , "dependencies" : { "requirejs" : "~2.1.5" , "jquery" : "~1.9.1" , "backbone" : "~0.9.10" , "underscore" : "~1.4.4" , "almond" : "~0.2.5" } }

.bowerrc 1 2 3 { "directory" : "app/components" }

1 $ bower install

Setup require.js configuration and main entry point. As I mentioned earlier, require.js supports modular development of both css and javascript. I went ahead and defined a main entry point for require.js, main.js, which includes configuration to use the libraries installed by Bower. I also stubbed out a simple Backbone view to test asset optimization later on.

app/scripts/main.js 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 requirejs . config ({ paths : { jquery : '../components/jquery/jquery' , backbone : '../components/backbone/backbone' , underscore : '../components/underscore/underscore' }, shim : { backbone : { deps : [ 'underscore' , 'jquery' ], exports : 'Backbone' }, underscore : { exports : '_' } } }); require ([ 'jquery' , 'marinara' ], function ( $ , marinara ) { 'use strict' ; var view = new marinara . SampleView (); $ ( 'body' ). html ( view . render (). el ); });

app/scripts/marinara.js 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 define ([ 'jquery' , 'backbone' , 'underscore' ], function ( $ , Backbone , _ ) { 'use strict' ; var exports = {}; var SampleView = exports . SampleView = Backbone . View . extend ({ id : 'sample' , template : _ . template ( '<h1><%= msg %></h1>' ), events : { 'click' : function () { alert ( 'Hello Shawn.' ); } }, render : function () { this . $el . html ( this . template ({ msg : 'Click me please.' })); return this ; } }); return exports ; });

Setup Grunt.js build system. To test changes during development, I setup an automated watch and reload workflow using Grunt. When files change in the app/ directory, I automatically reload the browser window with the LiveReload Chrome extension. When javascript files change in the / or lib/ directories, I restart the Node.js server using Upstart. Additionally, I also lint both css and javascript files when changes occur. To kickoff this workflow, I run grunt in the root directory. I also used Grunt to optimize static assets. The grunt-contrib-requirejs plugin uses require.js’s optimizer to concatnate and minify javascript. It also concatnate’s css by scanning for @import statements. I paired this with the grunt-contrib-cssmin plugin to minify css assets. To optimize assets and move them into the public/ directory to be served by Express, I run:

1 $ grunt optimize

Check out this gist for the Gruntfile.js that supports this setup. With the build system complete, the first exit criteria was met.

Implement Chef Application Recipe

With the Node.js project structure complete, I shifted back to the marinara-kitchen repository I worked on in the last post. I created a new recipe, application.rb, that would provision Node.js and related dependencies and deploy the marinara git repository. Below is the step-by-step guide to create the recipe:

Provision Node.js and npm packages. I leveraged the nodejs cookbook to install Node.js and npm by source defining the default versions in the default.rb attributes file. Additionally, I defined npm packages to be installed globally on the server (Grunt, Bower) using the LWRP provided by the npm cookbook.

site-cookbooks/marinara/attributes/default.rb 1 2 3 4 5 6 7 8 9 10 11 12 # includes nodejs default attributes first to override them include_attribute 'nodejs' # defines node.js and npm version configuration default . nodejs . version = '0.10.0' default . nodejs . npm = '1.2.14' # defines npm packages to install globally default . marinara . application . npm_packages = { 'grunt-cli' => '0.1.6' , 'bower' => '0.8.5' }

site-cookbooks/marinara/recipes/application.rb 1 2 3 4 5 6 7 8 9 10 # provisions node.js and npm include_recipe 'nodejs::install_from_source' include_recipe 'nodejs::npm' # provisions global npm packages node . marinara . application . npm_packages . each_pair do | pkg , ver | npm_package pkg do version ver end end

Provision Upstart script. Since I am using Ubuntu, I implemented an Upstart script to manage the Node.js application as a service. As I mentioned earlier, I configure the application using environment variables defined in the Upstart script injected in by a Chef template. The template also checks whether the application is being provisioned in production mode, and if so, includes statements to start the service on startup and run under an application-specific user account.

site-cookbooks/marinara/templates/default/marinara.conf.erb 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 description 'marinara upstart script' author 'Shawn Dahlen <shawn@dahlen.me>' < % if not node . marinara . application . development - %> start on [2345] setuid <%= node.marinara.application.user %> setgid < %= node.marinara.application.user %> env NODE_ENV= production < % end - %> env MARINARA_LOG_LEVEL=<%= node.marinara.application.log_level %> env MARINARA_PORT =< %= node.marinara.application.port %> stop on [06] respawn exec /usr/local/bin/node <%= node . marinara . application . deploy_path %>/server.js > <%= node . marinara . application . log_path %>/marinara.log 2> & 1

site-cookbooks/marinara/recipes/application.rb 1 2 3 4 5 # provisions upstart script template '/etc/init/marinara.conf' do source 'marinara.conf.erb' mode 0440 end

Deploy application. If the application will be provisioned in production mode (as opposed to development mode where Vagrant mounts the project directory), the recipe provisions the application user, creates the deployment and log directories, copies over a ssh deploy key, clones the marinara git repository, install dependencies, and optimizes assets. A number of default attributes drive the deployment — the most important of which is the reference attribute defining the git tag or branch to deploy.

site-cookbooks/marinara/attributes/default.rb 1 2 3 4 5 6 7 8 9 10 11 12 # defines marinara application deployment configuration default . marinara . application . development = false default . marinara . application . user = 'marinara' default . marinara . application . repository = 'git@github.com:smdahlen/marinara.git' default . marinara . application . reference = 'master' default . marinara . application . deploy_path = '/opt/marinara' default . marinara . application . log_path = '/var/log/marinara' default . marinara . application . log_level = 6 default . marinara . application . port = 8000 default . marinara . application . servers = { 'app0' => '127.0.0.1' }

site-cookbooks/marinara/recipes/application.rb 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 if not node . marinara . application . development # stops marinara service if running service 'marinara' do action :stop provider Chef : :Provider :: Service : :Upstart end # provisions system user to run application user node . marinara . application . user do system true shell '/bin/false' home '/home/marinara' supports manage_home : true end # provisions deploy key and ssh wrapper cookbook_file '/tmp/deploy_id_rsa' do source 'deploy_id_rsa' mode 0400 owner node . marinara . application . user end cookbook_file '/tmp/deploy_ssh_wrapper' do source 'deploy_ssh_wrapper' mode 0500 owner node . marinara . application . user end # provisions deploy directory with app user permissions directory node . marinara . application . deploy_path do owner node . marinara . application . user group node . marinara . application . user end # provisions log directory with app user permissions directory node . marinara . application . log_path do owner node . marinara . application . user group node . marinara . application . user end # provisions git repository git node . marinara . application . deploy_path do repository node . marinara . application . repository reference node . marinara . application . reference user node . marinara . application . user group node . marinara . application . user ssh_wrapper '/tmp/deploy_ssh_wrapper' end # provisions npm application dependencies execute 'npm install' do cwd node . marinara . application . deploy_path command '/usr/local/bin/npm install' user node . marinara . application . user group node . marinara . application . user env 'HOME' => "/home/ #{ node . marinara . application . user } " end # provisions bower application dependencies execute 'bower install' do cwd node . marinara . application . deploy_path command 'bower install' user node . marinara . application . user group node . marinara . application . user env 'HOME' => "/home/ #{ node . marinara . application . user } " end # optimizes browser assets execute 'grunt optimize' do cwd node . marinara . application . deploy_path command 'grunt optimize' user node . marinara . application . user group node . marinara . application . user end # starts marinara service service 'marinara' do action :start provider Chef : :Provider :: Service : :Upstart end end

Provision application firewall rule.To complete the second exit criteria, I defined a firewall rule in the application recipe to allow traffic on the specified application port with the upstream HAProxy server as the source.

site-cookbooks/marinara/recipes/application.rb 1 2 3 4 5 6 7 8 # provisions firewall rule to support incoming application traffic include_recipe 'simple_iptables' server = node . marinara . proxy . server port = node . marinara . application . port simple_iptables_rule 'application' do rule "-p tcp -s #{ server } --dport #{ port } " jump 'ACCEPT' end

Implement Chef Proxy Recipe

To scale the application servers horizontally with the growth of product usage, I implemented a Chef recipe to install and configure HAProxy. This recipe leveraged the haproxy cookbook for most of the heavy lifting. However, I had to reopen the template resource defined in that cookbook to use my configuration template instead. The code snippet below demonstrates this. In addition to providing custom configuration that references deployed application servers, I implemented a firewall rule to accept traffic on port 80 (the default value for the incoming_port attribute). In a future week, I intend to tune HAProxy’s configuration based on performance testing.

site-cookbooks/marinara/attributes/default.rb 1 2 3 4 5 6 7 8 9 10 # includes default haproxy cookbook attributes first to override include_attribute 'haproxy' # defines haproxy configuration default . haproxy . install_method = 'source' default . haproxy . source . version = '1.5-dev17' default . haproxy . source . url = 'http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev17.tar.gz' default . haproxy . source . checksum = 'b8deab9989e6b9925410b0bc44dd4353' default . marinara . proxy . server = '127.0.0.1'

site-cookbooks/marinara/recipes/proxy.rb 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # provisions haproxy include_recipe 'haproxy' # overrides haproxy cookbook default configuration template = resources ( 'template[/etc/haproxy/haproxy.cfg]' ) template . source 'haproxy.cfg.erb' template . cookbook 'marinara' # provisions firewall rule to support incoming http traffic include_recipe 'simple_iptables' simple_iptables_rule 'proxy' do rule "-p tcp --dport #{ node . haproxy . incoming_port } " jump 'ACCEPT' end

site-cookbooks/marinara/templates/default/haproxy.cfg.erb 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 global log 127 . 0 . 0 . 1 local0 maxconn 4096 user haproxy group haproxy defaults log global mode http option httplog option dontlognull option redispatch option forwardfor timeout connect 5 s timeout client 50 s timeout server 50 s balance < %= node.haproxy.balance_algorithm %> frontend http bind 0.0.0.0:<%= node . haproxy . incoming_port %> default_backend application backend application <% node.marinara.application.servers.each_pair do |name, addr| -%> server < %= name %> <%= addr %>:<%= node.marinara.application.port %> check < % end -%>

Test with Multi-Machine Vagrant Environment

To finish off the week, I created a Vagrantfile in the marinara-kitchen repository that creates three virtual machines using VirtualBox. Specifically, I created a definition for the HAProxy server and a definition for the Node.js application servers running on a private network so they could communicate with one another. Here is the configuration below using the new Vagrant 1.1 syntax:

Vagrantfile 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 app_servers = { app0 : '10.0.5.3' , app1 : '10.0.5.4' } Vagrant . configure ( '2' ) do | config | config . vm . box = 'precise64-chef11.2' config . vm . box_url = 'https://opscode-vm.s3.amazonaws.com/vagrant/opscode_ubuntu-12.04_chef-11.2.0.box' config . vm . provider :virtualbox do | vb | vb . customize [ 'setextradata' , :id , 'VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root' , '1' ] end config . ssh . forward_agent = true config . vm . define :proxy do | proxy | proxy . vm . network :private_network , ip : '10.0.5.2' proxy . vm . provision :chef_solo do | chef | chef . cookbooks_path = [ 'cookbooks' , 'site-cookbooks' ] chef . data_bags_path = 'data_bags' chef . add_recipe 'marinara::default' chef . add_recipe 'marinara::security' chef . add_recipe 'marinara::proxy' chef . json = { marinara : { application : { servers : app_servers } } } end end app_servers . each_pair do | name , ip | config . vm . define :" #{ name } " do | app | app . vm . network :private_network , ip : ip app . vm . provision :chef_solo do | chef | chef . cookbooks_path = [ 'cookbooks' , 'site-cookbooks' ] chef . data_bags_path = 'data_bags' chef . add_recipe 'marinara::default' chef . add_recipe 'marinara::security' chef . add_recipe 'marinara::application' chef . json = { marinara : { application : { reference : 'develop' }, proxy : { server : '10.0.5.2' } } } end end end end

With the Vagrantfile implemented, I brought the servers up and executed a quick test with curl against my endpoints. With a successful response, the fourth criteria had been met and the week’s task complete.