Announcing FoundationDB Document Layer

Today we’re open sourcing the FoundationDB Document Layer, a document-oriented database that extends the core functionality of the FoundationDB key-value store.

The FoundationDB Document Layer provides the ease-of-use of a document database in the form of the familiar MongoDB® API. The Document Layer is a stateless server, backed by the scalable and transactional features of FoundationDB.

Released under an Apache v2 license, the Document Layer project is hosted on GitHub and binary downloads are available for macOS and Linux. See the project documentation to get started, and we encourage your participation on the new forum section for the Document Layer.

MongoDB® API Compatible

For developers looking to use FoundationDB, the Document Layer has a familiar API, and is compatible with the MongoDB® protocol. In fact, simple applications leveraging MongoDB® can have a lift-and-shift migration to the Document Layer.

If you want to connect your application to the Document Layer, you can use any existing MongoDB® client — making it easy to get started. A discussion of the features supported by Document Layer can be found in the project's documentation.

Scalable Document Storage with FoundationDB

By extending FoundationDB, the Document Layer inherits key qualities of the core project — including years of testing, support for ACID transactions, and great performance. Using FoundationDB as a base for this stateless layer lets the Document Layer innovate in a number of notable ways.

No write locks: write operations, even on a single document, do not take locks. Transactions in FoundationDB are optimistic (lockless) and, by storing a document as multiple key-value pairs, the Document Layer supports many parallel writers. In some cases, operations may have to retry due to conflicts; however, non-conflicting operations, including metadata operations, can proceed in parallel.

No sharding: The Document Layer does not rely on a fixed shard key to distribute data. Instead, all data partitioning and rebalancing is managed automatically by the key-value store. This design, directly inherited from FoundationDB, provides robust horizontal scalability while avoiding client-level complexity.

Easy scaling: Each running instance of the Document Layer is a stateless application, configured only with the FoundationDB cluster to which it stores data. This stateless design means that many instances of Document Layer can be put behind a load balancer, all able to handle queries from any client and for any document.

Safe defaults: By default, write operations on the Document Layer execute with full isolation and atomicity. This consistency makes it easier to correctly implement applications that handle more than one simultaneous request. Indexes are always consistent with document data. Writes are fully consistent all the time irrespective of client selected options and, similarly, reads always see strong causal consistency.

Please see the documentation for more info about the architecture of the Document Layer.

The Future is Layers

When we first released FoundationDB we wrote in the original announcement:

The vision of FoundationDB is to start with a simple, powerful core and extend it through the addition of “layers”. The key-value store, which is open sourced today, is the core, focused on incorporating only features that aren’t possible to write in layers. Layers extend that core by adding features to model specific types of data and handle their access patterns.

The FoundationDB key-value store is powerful but, as noted above, its features remain narrowly scoped to distributed transactions and stateful storage. The Document Layer takes the key-value store's focused API and uses it to model a much more complex style of data storage. Because the Document Layer exposes a new, general-purpose API we think of this as an extension to FoundationDB.

The key-value store of FoundationDB lacks a number of features you would expect of a full database system. For instance there are no indexes, no data typing, and no query engine. The Document Layer extends FoundationDB to add a number of such concepts. Take index maintenance: key-value store transactions are used to simultaneously update both a document's field and an index value that points back at the field. This is a common pattern for how simple indexes can be built. As an extension to FoundationDB, Document Layer makes this pattern available to clients in a robust and reusable manner.

We could not be more excited to have the Document Layer as a community project! We're excited to see where we can take this together. Community collaboration does not need to end with the document model either — many other layers could be generally useful, written either as a library or a network server. The Document Layer should serve as a template for future layers both in design and in utility.

Disclaimer: MongoDB is a registered trademark of MongoDB, Inc..