Value Of Integration Testing

Outside of a few notable holdouts, most engineering teams recognize the immense value in testing software to improve code quality. Isolated unit-testing has become an increasingly common practice, with JVM libraries to help more complex unit-testing patterns such as dependency injection or mocking in testing libraries. While unit tests are valuable, they can’t gurantee a system works coherently.

There are fewer general libraries to help simplify integration and system testing (verifying interaction and end-to-end correctness). The reason why is pretty simple: it’s more challenging to do this kind of testing and harder to provide abstractions. You often need to instrument or mock multiple components and orchestrate several interaction steps (write your data to public cloud, wait to read results, send them somewhere else, etc). It’s relatively common for large frameworks to actually have special modules devoted to making it easier to do integration testing: Kafka, Spring, and Spark all have dedicated modules to allow for integration testing. However, there haven’t been general use case libraries to help with integration testing like we’ve seen with unit-testing.

At Amperity, we noted how cumbersome and repetitive it was to even setup the elements in an integration test. After some time in the hammock we decided the key thing missing was that testing needs to be composable. One of the reasons why we use Clojure for our micro-service codebase is because it makes it easy to build large-scale performant systems from small composable abstractions. We designed and open-sourced Greenlight to bring what we liked about writing production Clojure code to reduce the complexity of writing real-world integration tests.

Enter Greenlight

Greenlight is an open-source Clojure library for integration testing; it’s primarily used for “greenlighting” services to production. If you’re thinking to yourself the last thing the world needs is another testing library, you’d normally be right, but Greenlight is different. It’s not trying to replace assertions or even the syntax for how you specify tests. It’s fundamentally about allowing you to reuse and compose “testing steps” to more easily test complex systems and integrations.

Here’s a simple example of defining a test step. Cue the code sample:

;; throat clearing requires ( require ' [ greenlight.test :as test :refer [ deftest ]] ' [ greenlight.step :as step :refer [ defstep ]] ' [ greenlight.runner :as runner ] ' [ clojure.test :refer [ is ]]) ;; Greenlight tests are built from a collection of steps ( defstep math-test "A simple test step" :title "Simple math test" :test ( fn [ _ ] ( is ( = 3 ( + 1 2 ))))) ;; Use steps inside a test ( deftest simple-test "A simple test of addition" ( math-test ))

We’ll be reusing the require imports above so keep them fresh in your mental REPL. When you run the test you get snazzy terminal colors illustrating the steps and structure of your test.

Ok we’ve verified that arithmetic works, not so groundbreaking, but where Greenlight starts to shine is how you can connect steps together.

Step it up

To illustrate how Greenlight makes it easier to have complex test scenario, it’s worth going through a semi-involved example, since by design, Greenlight is made for complex testing scenarios.

For example, let’s say we maintain Hacker News and want to do a full test scenario that correctly calculates the number of times a link has been submitted. And by full test scenario, we mean end-to-end from creating users and having them submit links through the API to checking the resulting submission count.

In Greenlight, we’ll create steps for reusable testing logic using the defstep macro; here’s the reusable steps and the testing logic for the above scenario:

( defstep create-user "reusable step for creating user in test" :title "Create a temporary fake user" :inputs { ;; look up system component (see below) :api-client ( step/component :api/client ) ;; default value, but could use a test input from earlier :user-data { :handle "test42" :email "not@real.com" }} :test ( fn [{ :keys [ user-data api-client ]}] ;; call your API namespace functon to create user ( let [ response ( api/create-user! api-client user-data )] ;; common assertions ( is ( = 200 ( :status response ))) ;; register cleanup of user-id after test ;; (see below for definition) ( step/register-cleanup! :api/user ( :user-id response )) ( :user-id response ))) ;; name step output for downstream usage :output :user/id ) ( defstep submit-link "submit link and return submission id" :inputs { :user-id ( step/lookup :user/id ) :link "http://fake.com" :api-client ( step/component :api/client )} :title ( fn [ ctx ] ( format "User %s submits link" ( :user/id ctx ))) :test ( fn [{ :keys [ api-client, user-id, expected-submission-count, link ]}] ( let [ response ( api/submit-link! api-client user-id link )] ( is ( = 200 ( :status response ))) ( is ( = link ( :submission-link response ))) ( step/register-cleanup! :api/submission ( :submission-id response )) ( when expected-submission-count ( is ( = expected-submission-count ( :submission-count response )))) ( :submission-id response ))) :output :submission/id ) ( deftest multiple-submissions ( create-user ;; overwrite the default input { :user-data { :handle "user1" :email "user1@domain.com" }}) ( submit-link ;; picks up created user/id from previous step { :link "http://popular.com" :expected-submission-count 1 }) ( create-user { :user-data { :handle "user2" :email "user2@domain.com" }}) ( submit-link { :link "http://popular.com" :expected-submission-count 2 }))

That’s all the logic for the test itself. As we’ll see below, most of the above logic can be reused to test different scenarios, so we’ll get more value out of those defstep s. You might’ve noticed the step/component calls above to access the API client; this allows the tests to integrate with Stuart Sierra’s Component framework, as we’ve found it a good way to construct service resources. We pass the system components (if you require on for your tests) into the test runner:

( require ' [ com.stuartsierra.component :as component ]) ;; Main runner ( defrecord ApiComponent [] component/Lifecycle ( start [ this ] ( println "Starting API client" ) ;; configure client this ) ( stop [ this ] ( println "Stopping API client" ) ;; teardown this )) ( runner/run-tests! ;; steps use step/component to fetch system components ( constantly ( component/system-map :api/client ( ->ApiComponent ))) ;; fetch all tests to run in project [( multiple-submissions )] {})

The only thing missing from the above to run this full suite of tests are knowing how to do the cleanup work hinted at via step/register-cleanup! . Cleaning state is an important element of complex testing, particularly when it may result in actual database entries and other stateful changes. The cleanup system uses Clojure multi-methods to allow for extensibility. In order to utilize this system, you only need to define a generic cleanup method for each type of resource, which will get passed the test system and whatever you pass at the register-cleanup! call-site. For the test above, you would do the following:

( defmethod step/clean! :api/submission [ system _ submission-id ] ;; cleanup DB state ) ( defmethod step/clean! :api/user [ system _ user-id ] ;; delete user )

Now that we’ve got all the pieces we can actually run the test and we’ll get some pretty looking terminal output, which makes it easy to understand exactly where a failure may have happened.

Now if all we were testing were the above scenario, the code, while more concise, isn’t an order-of-magnitude shorter than just a monolithic test. But consider if we actually try to provide coverage over multiple scenarios that reuse some of these steps (creating users, submitting links, etc.). For instance, we probably should also test that after a user submits a link, we should be able to retrieve it for the user profile and be the most recent submission. Here’s what extra code you need to test that:

( defstep fetch-submissions "fetch submissions for user sorted by recency" :inputs { :user-id ( step/lookup :user/id ) :api-client ( step/component :api/client )} :test ( fn [{ :keys [ user-id api-client ]}] ( let [ response ( api/fetch-submissions api-client user-id )] ( is ( = ( :status response ) 200 ) ( :submission-ids response )))) :output :submission/ids ) ( deftest find-recent-submission ( create-user ) ( submit-link ) ( fetch-submissions ) ;; inline step since not reused # ::step { :name ` check-submitted-link :title "Check submitted link is most recent" :inputs { :target-submission-id ( step/lookup :submission/id ) :submitted-ids ( step/lookup :submission/ids )} :test ( fn [{ :keys [ target-submission-id, submitted-ids ]} ;; target submission should be most recent ( is ( = ( first submitted-ids ) target-submission-id ))])})

A Library of Composable Fixtures

If you’ve used enough testing libraries, including Clojure’s own clojure.test , you might recognize that the defstep plays a role somewhat like test fixtures (functions which execute before/after an actual test). This is broadly correct, except that defstep s are composable across a code-base via inputs and outputs, rather than being namespace isolated. They of course also combine the ability to add test assertions for common cases, which add more compositionality to the set of assertions for a given test scenario.

This flexibility is really valuable for a complex set of (micro)services, where many components can be used in several places. You can reuse the step logic for writing to S3, or dealing with database reads/writes, etc. The end result is that your testing code gets much of the composability you’ve come to expect in your non-testing Clojure code.

Sneaking Greenlight Into a Large Java Project

If you’re reading this and have been Clojure curious, Greenlight might be the perfect way to dip your toes in the parenthetical Clojure waters. One of Clojure’s greatest strengths is seamless java interop, which would allow you to use Greenlight to test a Java project. This is a low-risk way to mix Clojure in to your existing JVM codebases and get immediate value.

At Amperity, we built Greenlight in order to bring some of what makes Clojure great for writing software (small highly-reuseable and composable functions) to our testing strategy. So if you enjoy what Greenlight brings to your testing, imagine what your codebase would be like if it was entirely in Clojure.