The DevOps Recipe

Nailing down the DevOps process around your consumers and providers pipeline is crucial because that will be the key factor to a successful adoption of Consumer-Driven Contract testing in your organization.

Several of those key factors to take into account are:

CI and testing strategies — A good understanding of the testing strategy in your organization and an already established build and CI pipeline that you can tap into.

Source Code Versioning workflow — The Pact framework and Consumer testing is agnostic to whichever source code versioning you use, and the workflows you practice. With that said, the processes of running your contract pipeline, verification and go-to-production practices will vary depending on whichever workflow you adopt.

A Trunk Based Development Git Workflow

Disclaimer: the Pact framework docs assume a git flow practice where feature branches are used on both consumer and provider teams to contain work-in-progress development of a contract, and then merged once the contract is verified.

This guide is based on a trunk based development workflow, at least for the consumer team. This means that consumer implementations and contracts are assumed to always be developed and available on the master branch, even if their implementations by providers are not yet available or existing. This is an important bit because the Pact Manifest concept that is introduced in this guide is largely required due to this workflow.

This affects the go-to-production practice and introduces the use of a Pact Manifest facility, but the rest of the guide and DevOps process mentioned through-out is still applicable regardless.

Trunk-Based Development is a key enabler of Continuous Integration and by extension Continuous Delivery. src: https://trunkbaseddevelopment.com

A trunk based development workflow resembles how many Open Source projects are handled — there’s a master branch that streamlines the main development branch, and any pull requests made to introduce a fix or a feature are usually short lived and merge into the main-line development.

The notion of feature, bug, release, or hotfix branches are non-existent, and instead of long-lived branches there’s a more pressing attitude of continued code integration into the master branch.

This may raise some questions, for example — how can I merge consumer code and its contract test where the provider hadn’t yet implemented it?

Even though some work-in-progress code paths exist in production, it doesn’t mean they get met. Using feature toggles allow to enable/disable a code path based on whether a functionality is ready, or exposing it to specific/limited users only.

API versioning is another facility that can be used to easily expose a work-in-progress code path , that may not even function, but can be tested and developed for until it is ready, and once its ready releasing it under a new API version. This strategy allows maintaining API paths without breaking contracts, or at least until users of the API had enough reasonable time to upgrade to the new version.

Consumer-Provider Interaction

Consumers will engage in creating the pacts (the API contracts) and push them to the Broker, where the pacts can then be downloaded by Providers to run tests against them. When provider tests fail, the provider’s CI will prohibit the provider from moving on to production, as indeed that will break a contract with its consumers.

The flow depicted in the diagram is as follows:

A consumer will generate the pact contracts and publish them to the broker (upon a release, as in, merge to master branch for example). The default state and assumption is that the API contract isn’t yet implemented by the provider, so the consumer has no interest in breaking the provider’s build and will tag the contract as dev when it is publishing the contracts to the broker. The provider in its CI tests (whether that runs on a Pull Request or on a master branch before deploying), will download all the consumer contracts that are tagged as prod and perform its side of provider testing to ensure that the provider code is not breaking the contracts.

At this point there aren’t any prod tagged contracts so CI continues as normal. At a later point in time when the Provider has implemented the contract, and it is now available in production, the consumer will pro-actively tag that contract as prod (manually, yet can be automated with custom tooling) and publish it to the broker. The next provider CI tests that run will now pull in this contract that is tagged as prod. If the provider tests fail, they should fail the CI and the whole build process to prohibit the provider from moving on with the change (as this will break its consumers). Regardless of the verification outcome in the provider testing, such as failure or successful tests, this information is reported back to the broker.

General notes about the above flow:

When the consumer tags the contract as prod it also relies on confirming that the contract has been verified successfully in the last CI run of the provider and so that it is indeed ready on the provider side to be tagged as such. If it were tagged as prod by the consumer and published to the broker without the provider actually implementing the functionality, it would cause the provider CI to fail. Pact’s can-i-deploy tool can be used to mitigate this by only allowing to tag contracts as prod when they been verified at least once.

it also relies on confirming that the contract has been verified successfully in the last CI run of the provider and so that it is indeed ready on the provider side to be tagged as such. If it were tagged as by the consumer and published to the broker without the provider actually implementing the functionality, it would cause the provider CI to fail. Pact’s can-i-deploy tool can be used to mitigate this by only allowing to tag contracts as when they been verified at least once. The provider CI will download and run both dev and prod contracts and will fail only when the prod contracts test fail. This allows to gain visibility on the work-in-progress made on the provider side with regards to the dev contracts.

The Consumer’s DevOps Pipeline

Taking a deeper look into the mechanics of the Consumers pipeline will further shed a light on when contract testing takes place, and what is the interaction like with the broker and pact manifest tagging.

Consumer DevOps pipeline

Build

In the life-span of a Pull Request we run several tests during the build phase and take care of publishing a so-called build contract that is only used in order to download it later in the CI phase.

In the build phase we are basically pulling the code over from GitHub, running basic static code style & quality analysis commonly known as linting, then run unit tests, contract tests and integration tests. Contract tests are run just like unit tests but they run as another build task just to differentiate.

The integration tests we refer to in the build phase are about less code mocking and stabbing, and more about covering code paths that integrate with other components, such as triggering an API controller that consequently hits the database (so we spin up a database, or a mocked database like a Druid mock service).

When tests are successful a contract is published to the broker, so it can be downloaded in the CI step ahead to be used in End-to-End tests as mocked providers.

CI

What the aforementioned integration tests are not — they are not spinning up a full fledged deployed system that run an End-to-End functional test. This happens in the CI.

In the CI stage we setup an environment that replicates production as much as possible, but as per the spirit of consumer-driven contracts, we don’t spin-up real instances of API providers. We use docker-based containers and spin-up consumer containers that were created in the build stage before-hand, as well as spin-up any relevant databases that are direct dependencies of the consumer.

All the providers that the consumer needs to interact with are created by downloading the contracts and spinning-up mocks using utilities from the Pact framework. The mocks respond with canned responses based on how the contract was created.

We leverage CDC testing in the E2E phase where we can benefit from not requiring to bring up real third-parties and can test our application/service in end-to-end flows on a semi-real environment.

Master

Once code has been successfully tested and merged into the main line of development — the master branch, it may see more tests, but mainly what happens is:

The service version is incremented and gets tagged.

Contracts are published to the broker (by default using the dev tag for newly introduced contracts).

The Pact Manifest

The Pact Manifest is an internal tool I developed during my time with Nielsen Marketing Cloud to completely de-couple the consumer and provider without having to rely on branching conventions to keep them synced.

The pact manifest is a simple JSON file that resides on the consumer’s code-base with the role of tracking the state for a consumer-provider contracts in terms of, whether a contract is still a work-in-progress or haven’t been released to production which will be tagged as dev or the contract has been implemented by a provider and is deployed and available in production where-as the contract will be tagged as prod.

The pact manifest is needed because the source code versioning workflow set out by the teams is a trunk based development one. Which means that if an integration with provider is developed in a branch, including all the relevant consumer contract tests, it is very quickly merged to the master branch after being reviewed and passing all the tests in that Pull Request branch. It is merged to master even if the provider didn’t start implementing anything.

This is where the pact manifest is useful. It defaults to tagging the contract as dev to begin with, and later on when the consumer received the update that the provider had implemented and released the API a manual update is made to the pact manifest file to tag that contract as prod and publish it to the broker as such.

It’s important to note that it can be a mistake on the consumer side to tag a contract as prod and publish it because in reality the provider didn’t yet finish implementation or its implemented but not yet released to production. When this happens it will cause the provider CI to fail and block that team from releasing to production.

To mitigate this scenario:

Ensure proper communication is made between the teams. There is literally no excuse or replacement for good collaboration regardless of consumer-driven contracts or otherwise.

The pact manifest uses a little gem from the Pact framework called can-i-deploy to verify whether a contract can be tagged as prod.

The can-i-deploy tool is another key element in the process. It pings the broker to check if a consumer’s contract was verified by the provider. If it hasn’t, it’s a good sign that you shouldn’t deploy or tag a contract as prod-ready because it isn’t yet implemented by a provider.

The Provider’s DevOps Pipeline

The provider contract testing is all about confirming that it didn’t break any API contract it has with it’s consumers.

To do that, we need to spin up a real instance of the provider’s API service, preferably with a real database as most probably it will need to mutate state. This means that it’s most suitable to run the provider contract testing in CI so the testing is closer to an integration or end-to-end test.

Once the provider API service is up, it is also required to make available an endpoint that can be called by the pact runners before each interaction is being performed in order to tell the provider which state to change to.

Provider DevOps pipeline

In essence, when the CI runs, the consumer contracts are fetched from the broker, and the provider API service is brought up to serve requests. The pact framework will read each consumer contract and start playing the requests one by one, before each of them it will advise the provider what state it is expected to be, and once the request has been sent, the pact framework will verify the interaction.

Once all the interactions have ran, upon both a successful or failed contract testing session the results will be sent back to the pact broker where they are recorded for inspection and visibility into this insight.

The DevOps Recipe — Putting it all together

The following is a sequence diagram depicting the entire flow across the Consumer, Broker and Provider.