Thorough testing continues to be a key tenet of software development at clypd. Quality is paramount, but shipping code quickly requires us to be efficient in how we search for bugs. We’ve found innovative ways to unit test individual Go packages of our programmatic television ad platform as it has grown in its capabilities. The construction of a similarly efficient mechanism to test large portions of the whole system has taken our test automation to the next level.

Integration testing goals

Integration tests can be expensive to implement and run. In order to get the most out of the effort invested, an integration testing system has to make writing and executing tests as easy and painless as possible. We set out to build a system with the following properties:

1. Easy to add new components – As our ad platform evolves, the addition of new components and third-party systems to our platform’s ecosystem is expected. These should be made trivial to add to our integration environment.

2. Easy to repeat – Integration tests should be easy to execute on demand. Even if something goes wrong and a test execution fails catastrophically, executions that follow should start with a clean slate.

3. Fast – Many integration test environments I have worked with previouslyhave exhibited such poor runtime performance as to be not feasible to run on every check-in. This undermines the value of the entire suite, so serious effort must be made to ensure setup, execution and teardown run as quickly as possible.

4. Portable – All developers are responsible for building out the test suite along with features they work on, so the integration environment must run consistently on both local development machines and continuous integration builders. In other words, developers shouldn’t need access to CI to develop for it.

Enter Docker

Docker container abstractions fit in well with our integration testing approach. Many of Docker’s design philosophies for structuring deployments of clustered applications to the cloud work very well for our more modest goals.

Preferring environments composed of Docker containers over monolithic VMs running all components makes the approach for mixing in new components obvious: just spin up a Docker container for it and link it to other containers that need it. Once Docker images are built, startup and teardown are extremely fast. This makes starting from a clean slate on each test iteration a breeze. Finally, Docker itself has few requirements other than Linux. Once Docker is up, environments running within Docker containers are completely portable. The entire test suite can be run equally well on OS X laptops running Vagrant as it can on native Linux.

Our first Docker integration tests rely on just two containers: one to run Postgres and another to test our Go source code with the Postgres service. We chose CoreOS as the host operating system in both CI and locally because it ships with Docker installed and focuses on very little other than providing an ideal, unencumbered environment for running containers. A Vagrantfile for CoreOS is checked into our git repository to provide this environment for developers running Macs.

With Docker itself up and running, starting up a Postgres service is easy enough:

docker run --name postgres-db -d postgres:9.3.4

This starts up a container that exposes the Postgres daemon on its default port: 5432. For our Go container, a custom image is constructed using a Dockerfile checked in with the source code.

FROM ubuntu:14.04 RUN apt-get update && apt-get -y install golang make ENV GOPATH /var/app RUN mkdir -p $GOPATH # run-integration-test.sh sets up a config file for the app, then invokes ‘go test’ CMD $GOPATH/run-integration-test.sh ADD ./ $GOPATH

This file starts with a base Ubuntu image, installs Go and Make (which orchestrates the setup and execution of tests in the container), sets the default command to run integration tests and finally copies in our source code. The ordering of these steps is important. Specifically, Docker will use previously cached step execution results rather than executing from scratch if it detects that nothing important in step execution context has changed since last time. Normally, every step up until the last ADD which picks up source code changes can be cached. This caching mechanism enables image build times of just a few seconds.

The Go image is built and executed as a container with the Postgres container linked to it:

docker build -t go_app/latest . docker run --name go-app --link postgres-db:postgres-db go_app/latest

The –link parameter works behind the scenes to expose the Postgres service’s IP address and port to the Go container. Go can discover the Postgres service through the system environment, e.g.

printenv POSTGRES_DB_PORT_5432_TCP_ADDR

The structure of this environment variable seems very specific, but is in fact constructed by Docker according to the linked container name (postgres-db), ports exposed (5432) and protocol (TCP). This simple mechanism is perfectly serviceable for informing containers about peers they must communicate with. The logic in run-integration-test.sh mostly consists of environment reads of the variables necessary to configure the application for integration testing.

Additional service components can be added following the same principles. New container daemons are either run directly off of images from DockerHub, or are built from Dockerfiles, then executed. Once up and running, they are added as new –link parameters to the Go integration test container.

Now test setup in Go

With the infrastructure ready to roll, a few useful constructs are used in our Go tests to make integration testing as straightforward as possible. We rely on the same ‘Go test’ utility for both integration and unit tests. Go’s ‘testing’ package has several features which are especially useful for integration testing:

T.Skip() – Separating the unit and integration test suites is important to keep our build process nice and tidy. The ‘testing’ package’s T.Skip() is combined with Go’s flags in order to omit integration tests from the unit test suite

integrationEnabled = flag.Bool("integration", false, "enable integration tests") func Test_SomeIntegrationTest(t *testing.T) { if !*integrationEnabled { t.Skip("skipping integration test") } // … the rest of the test }

The ‘-integration’ command line flag is enabled by our integration test runner, but is otherwise omitted by unit tests.

Transactions for database cleanup – The same Postgres container is reused for the entire test suite, so it’s important for tests to clean up after themselves. Postgres transactions make this easy. Each test starts a new transaction, sets up data, then performs and verifies its assertions within that transaction. During test teardown, the transaction is rolled back. This process is repeated for each test.

At clypd, we use gorp and pq for our interactions with Postgres. An interface that mirrors most of gorp’s CRUD functions has been defined to facilitate transactions. All database interaction in our code occurs through this interface, for which transactional and non-transactional implementations have been built. This allows tests to easily substitute a transactional implementation for the standard one that would otherwise be used. This is one of many examples where Go’s interfaces have provided valuable flexibility to our code. Here’s a simplified example of how this interface works for gorp’s Insert and AddTableWithName functions:

type Datastore interface { AddTableWithName(interface{}, string) *gorp.TableMap Insert(...interface{}) error } // the embedded gorp.DbMap provides a Datastore-compatible Insert type gorpDatastore struct { gorp.DbMap } // just a wrapper around gorp’s version func (ds *gorpDatastore) AddTableWithName(type interface{}, table string) *gorp.TableMap { return ds.DbMap.AddTableWithName(type, table) } // the embedded gorp.Transaction provides a Datastore-compatible Insert type gorpTransaction struct { *gorp.Transaction gorpDS *gorpDatastore } // provides the functionality by delegating to the gorpDatastore associated with the transaction func (ds *gorpTransaction) AddTableWithName(type interface{}, table string) *gorp.TableMap { return ds.gorpDS.AddTableWithName(type, table) }

In this example, the transaction’s AddTableWithName delegates to the parent non-transactional implementation. This design pattern allows clients to remain agnostic about whether they are dealing with a transactional or non-transactional data store. Clients work solely through the interface.

One complication we ran into while adapting our code to use the newly defined interface relates to our use of Prepare(query) as part of COPY queries for bulk data loading. ‘Prepare’ was not provided by gorp, so we had originally gone a level lower to access the function from the database/sql API. We recently contributed a patch to gorp that enables this functionality. With this change in place, both the transactional and non-transactional implementations can perform all of their functions using the gorp API.

Pass in *testing.T to fail fast – Keeping the extra setup/teardown boilerplate that creeps into integration tests to a minimum is important to maintain simple, easily comprehensible tests. A pattern that we find useful to decrease verbosity is to pass the *testing.T test state manager into utility functions that can use it to fail fast, rather than returning errors for calling tests to check for. For example, given a basic type used as part of integration tests:

type Account struct { ID int Name string }

A first draft at a function to use a gorp DbMap to insert Accounts to facilitate testing may look like:

func InsertAccount(ds Datastore, a *Account) error { ds.AddTableWithName(Account{}, "accounts").SetKeys(true, "id") return ds.Insert(a) } // in tests, using ‘assert’ from github.com/stretchr/testify err := InsertAccount(ds, &Account{1, “foo”}) assert.NoError(t, err)

This works, but tests are responsible for checking for returned errors. Considering this is a function for setting up prerequisite data, this check is seldom relevant to the main crux of whatever is being tested. Also, this assertion will not abort the test if data setup fails. An improved version of InsertAccount solves these problems:

func MustInsertAccount(t, *testing.T, ds Datastore, a *Account) { ds.AddTableWithName(Account{}, "accounts").SetKeys(true, "id") if !assert.NoError(t, ds.Insert(a)) { t.FailNow() } } // in tests MustInsertAccount(t, ds, &Account{1, “foo”})

Now, persisting an Account from a test requires just one function call. The error check happens behind the scenes and will abort the test on failure.

Looking forward

Building out integration tests has allowed our team to spend time testing far more efficiently. We find ourselves catching database bugs much earlier in the development process, when they’re easier to diagnose and fix. We’re building out a regression test suite which can be executed against our platform extremely quickly. Best of all, we’re spending far less time on tedious manual testing.

Getting our feet wet with Docker for this project has us excited for container technology on a wider scale. We are currently exploring using containers to reduce the footprint of our development sandbox environments. Stay tuned for another blog post about our progress with Docker.