Testing Linearizability in MongoDB 3.4

Jepsen is a tool developed by Kyle Kingsbury, aka Aphyr, for testing how databases perform in the face of network partitions. Kyle uses Jepsen to test distributed systems, including MongoDB, and publishes the outcomes on his blog. Since he started his “Call Me Maybe!” series in 2013, Kyle has tested over 20 distributed systems with Jepsen, and his rigorous and detailed writeups are fantastic. In May 2013, he posted an analysis of MongoDB 2.4.3 with respect to linearizability. (Spoilers: at that time, MongoDB did not support linearizable reads.)

As we introduced support for linearizable reads in MongoDB 3.4, we wanted to ensure robust testing for this feature. Jepsen, the gold standard in testing linearizability, was obviously the right tool for the job. We started by contracting with Kyle to test MongoDB 3.4 extensively prior to release.

Kyle’s write-up of his results is now available, but the real goal of our collaboration with Kyle was to integrate Jepsen with our automated testing framework. At MongoDB, we run all tests in Evergreen, our continuous integration (CI) system. Running automated tests in Evergreen enables us to verify improvements and prevent regressions. Running the Jepsen tests in Evergreen gives us confidence that linearizable reads are operational and that no regressions have been introduced as we continue to develop MongoDB. In order to integrate Jepsen with Evergreen, we needed to modify Jepsen to support the Evergreen environment. In the rest of this article, I'll provide an overview of Jepsen and Evergreen and walk through the process of integrating the two systems.

Jepsen Overview

Jepsen tests distributed databases for consistency by using several clients to independently perform compare and set (CaS) operations. CaS operations involve reading and writing integers to the database. A CaS operation performs an update to a record contingent on its current value, and the resulting value is expected to be available to all clients reading that record. Each write, acknowledged by the database, is recorded in a separate log. After the clients complete, Jepsen checks the databases against the recorded results in its log using the Knossos analysis tool. Knossos uses an array of techniques to analyze a history of a writes for linearizability. If either acknowledged writes are missing or unacknowledged writes are present, then the database is considered inconsistent with respect to the clients. This inconsistency would prove that the database did not handle the network partition properly.

Jepsen tests are broken up into a series of phases: cluster setup, workload generation, chaotic network partitioning, and application analysis. I'll provide an overview of each phase as background for the challenges we faced in integrating Jepsen with our testing framework.

Cluster Setup

The cluster setup starts up a MongoDB replica set. As seen in the following diagram, the database runs on 5 nodes, named n1 through n5, with n1 started as the primary node. Note that the Jepsen control node resides on a separate node, which can always communicate with all the nodes in the cluster.

The cluster is implemented using Linux containers, such that each container hosts one mongod process of the replica set.