We recently shared the story of our transition to Node.js at Node Summit. As part of that transition, we developed an integration testing tool called testium.

We’d like to announce that testium is now open source!

What is testium?

With testium you can write integration tests that:

use a familiar BDD syntax

are written in CoffeeScript/JavaScript

have a synchronous API

can leverage Selenium

Why Create a New Tool?

Since the beginning of our transition to Node.js, we have been looking for a great way to include integration testing in our Node.js applications. In Ruby on Rails, we had a Cucumber setup that could run tests in real browsers via Selenium. The relationship with testing tools was not strongly defined for Node.js (the platform) or even Express (the underlying framework).

That lead us to look for other existing tools we could adopt. We tried a couple of tools, but they just didn’t fit our needs for different reasons.

Exploring WebDriver

We wanted to write tests in JavaScript and Mocha with a synchronous API that takes advantage of our existing Selenium infrastructure. WD.js came close, but the challenge for full adoption came with the syntax, which is callback-based or promise-based. These are typical patterns in JavaScript projects, but for an integration testing tool, we only want to execute serial actions. A synchronous API made the most sense.

A spike seemed to be in order. We read up on the WebDriver spec and started implementing Node.js bindings that used http-sync to get synchronous http. The spike used Mocha to run the tests and provided an API that executed WebDriver calls against a local Selenium Standalone Server. This approach seemed to be so beneficial for us that we decided to move forward with it.

testium

testium is the result of that effort. Here are a few examples.

Navigation is easy:

{getBrowser} = require 'testium' assert = require 'assertive' describe 'browse', -> before -> @browser = getBrowser() @browser.navigateTo '/browse/abbotsford' assert.equal 'status code', 200, @browser.getStatusCode() it 'shows deals', -> deals = @browser.getElements '.deal' assert.equal deals.length, 20

DOM Interaction can be done in a couple of ways:

{getBrowser} = require 'testium' assert = require 'assertive' describe 'browse', -> before -> @browser = getBrowser() @browser.navigateTo '/browse/chicago' assert.equal 'status code', 200, @browser.getStatusCode() it 'allows a user to search near a different division', -> query = @browser.getElement '#search' query.type 'pizza

' heading = @browser.getElement '.browse-title-heading' text = heading.get 'text' assert.equal 'pizza', text

Cookies can be set before making requests:

{getBrowser} = require 'testium' assert = require 'assertive' describe 'browse', -> before -> @browser = getBrowser() it 'shows a logged-in user', -> userId = 'dd300b20-578f-11e3-872b-0002a5d5c51b' @browser.setCookie name: 'user_id' value: encodeURIComponent(userId) @browser.navigateTo '/browse' assert.equal 'status code', 200, @browser.getStatusCode() @browser.assert.elementHasText '.user-name', 'Some User'

Screenshots can be taken and (experimentally) diffed:

{getBrowser} = require 'testium' assert = require 'assertive' describe 'screenshots', -> before -> @browser = getBrowser() @browser.navigateTo '/' assert.equal 200, @browser.getStatusCode() it 'can be taken', -> data = @browser.getScreenshot() assert.truthy data.length > 0 it 'can be compared', -> screenshot1 = @browser.getScreenshot() screenshot2 = @browser.getScreenshot() @browser.assert.imagesMatch(screenshot1, screenshot2)

Failures automatically take screenshots for you:

Testing against: phantomjs screenshot 1) exists 0 passing (2s) 1 failing 1) exists Error: Assertion failed: screenshot exists Expected: true Actually: false [TESTIUM] Saved screenshot {app}/test/integration_log/screenshots/exists.png at error ({app}/node_modules/assertive/lib/assertive.js:365:12) at Object.assert.truthy ({app}/node_modules/assertive/lib/assertive.js:59:15) at Context. ({app}/test/integration/dialog_test.js:29:23) at Test.Runnable.run ({app}/node_modules/mocha/lib/runnable.js:211:32) at Runner.runTest ({app}/node_modules/mocha/lib/runner.js:355:10) at Runner.runTests.next ({app}/node_modules/mocha/lib/runner.js:401:12) at next ({app}/node_modules/mocha/lib/runner.js:281:14) at Runner.hooks ({app}/node_modules/mocha/lib/runner.js:290:7) at next ({app}/node_modules/mocha/lib/runner.js:234:23) at Runner.hook ({app}/node_modules/mocha/lib/runner.js:253:7)

You can see that a screenshot has been saved “[TESTIUM] Saved screenshot {app}/test/integration_log/ screenshots/on_failure.png”.

Testing Stack Consistency

One great benefit of this setup is that our test suites are now very similar:

Server Unit Tests: Mocha + Bond + Assertive

Client Unit Tests: Mocha + Bond + Assertive

Integration Tests: testium (Mocha) + Assertive

Limitations

The philosophy of WebDriver is that it should allow you to programmatically act like a user, but it does have limitations we have to work around. As a result there are some features that the maintainers will not implement because a user would not be able to exercise that feature autonomously.

Further, there are existing technical limitations due to WebDriver being a common standard across all browsers. Some aspects can’t be implemented in a specific browser and therefore these features are not implemented at all.

There are four specific issues with WebDriver we wanted to resolve:

response status codes unavailable

response headers unavailable

request headers not modifiable

page must be loaded to set cookies

Details on our workarounds will come in a follow-up post.

Going Forward

testium is already serving our needs, but we have some features planned for the future. Development will continue on the GitHub repository. Check out the Roadmap and let us know what you’d like to see by creating issues. Pull requests are always welcome!