Recently I’ve been testing clusters of VMs running on my local host.

I thought that there must be a standard way to test multi-node VM setups, but asking around at work, and on github yielded no answers.

So I came up with my own solution, which I outline here.

ShutItFile

A ShutItFile is a superset of a Dockerfile that allows straighforward automation of automation tasks.

Here’s an example of a ShutItFile that manipulates two VMs, and tests network connectivity between them.

It creates two machines (machine1 and machine2) and logs into them in turn using the ‘VAGRANT_LOGIN’ directive. On each machine it installs python, sets up a simple python http server which serves the text: ‘Hi from machine1’ (from machine1) or ‘Hi from machine2’ from machine2.

It then tests that the output matches expectation from both machines using the ‘ASSERT_OUTPUT’ directive.

To demonstrate the ‘testing’ nature of the ShutItFile, a ‘PAUSE_POINT’ directive is included, which drops you into the run with a terminal, and a deliberately wrong ‘ASSERT_OUTPUT’ directive is included to show what happens when a test fails (and the terminal is interactive). This makes debugging a _lot_ easier.

DELIVERY bash # Set up trivial webserver on machine1 VAGRANT_LOGIN machine1 INSTALL python # Add file RUN echo 'hi from machine1' > /root/index.html RUN nohup python -m SimpleHTTPServer 80 & VAGRANT_LOGOUT # Set up trivial webserver on machine2 VAGRANT_LOGIN machine2 INSTALL python RUN echo 'hi from machine2' > /root/index.html RUN nohup python -m SimpleHTTPServer 80 & VAGRANT_LOGOUT # Test machine2 from machine1 VAGRANT_LOGIN machine1 INSTALL python RUN curl machine2 ASSERT_OUTPUT hi from machine2 VAGRANT_LOGOUT # Test machine1 from machine2 VAGRANT_LOGIN machine2 INSTALL python RUN curl machine1 ASSERT_OUTPUT hi from machine1 VAGRANT_LOGOUT # Example debug VAGRANT_LOGIN machine1 INSTALL python PAUSE_POINT 'Have a look around, debug away' # Trigger a 'failure' RUN curl machine2 ASSERT_OUTPUT will never happen VAGRANT_LOGOUT

To run this ShutItFile (which we call here ‘ShutItFile.sf’), you run like this:

# Install shutit pip install shutit shutit skeleton --shutitfile ShutItFile.sf \ --name /tmp/shutitfile_build \ --domain twovm.twovm \ --delivery bash\ --pattern vagrant \ --vagrant_num_machines 2\ --vagrant_machine_prefix machine

This code for this example is available here.

Video

There’s a video of the above run here:

Create Your Own

If you want to create your own multinode test:

pip install shutit #use sudo if needed, --upgrade if upgrading shutit skeleton

Follow the instructions, choosing ‘shutitfile’ as the pattern, and ‘vagrant’ as the delivery method, eg:

$ shutit skeleton # Input a name for this module. # Default: /space/git/shutitfile/examples/vagrant/simple_two_machine/shutit_sabers # Input a ShutIt pattern. Default: bash bash: a shell script docker: a docker image build vagrant: a vagrant setup docker_tutorial: a docker-based tutorial shutitfile: a shutitfile-based project (can be docker, bash, vagrant) shutitfile # Input a delivery method from: bash, docker, vagrant. # Default: ' + default_delivery + ' docker: build within a docker image bash: run commands directly within bash vagrant: build an n-node vagrant cluster vagrant # ShutIt Started... # Loading configs... # Run: cd /space/git/shutitfile/examples/vagrant/simple_two_machine/shutit_sabers && ./run.sh # to run. # Or # cd /space/git/shutitfile/examples/vagrant/simple_two_machine/shutit_sabers && ./run.sh -c # to run while choosing modules to build.

and follow the commands given (at the place in bold above) to run.

Initially you are given empty ShutItFiles. You could start by adding the commands from the example here.

A cheatsheet for the various ShutItFile commands is available here.

Watch me do this here.

Real-world Usage

As an example of real-world usage, this technique is being used to regression test Chef recipes used to provision OpenShift.

The Chef scripts are here, and the regression tests are here.