FoundationDB 6.2.7 Released FoundationDB 6.2.7 is now officially available! This release focuses on improving the overall performance and reliability of the database. The latest FoundationDB release can be downloaded and installed as binaries from the downloads page (available for macOS, Windows, Linux), or as source from our GitHub repository. If you're already running FDB, also see our upgrade instructions. Some highlights from this release include: Improved read throughput : Performance improvements to the SSD storage engine increased read throughput by over 25%.

: Performance improvements to the SSD storage engine increased read throughput by over 25%. Cheaper client connections : The 6.2 release significantly reduces the impact connected clients have on clusters. These changes also improve the overall scalability of FoundationDB, because clients no longer keep connections open with the cluster controller.

: The 6.2 release significantly reduces the impact connected clients have on clusters. These changes also improve the overall scalability of FoundationDB, because clients no longer keep connections open with the cluster controller. Smarter data distribution: FoundationDB’s method of balancing data across servers now does a better job avoiding write hotspots. A full list of features, fixes, and other changes are documented in our release notes.

FoundationDB Summit 2019 Program Announced We’re pleased to announce the program for the FoundationDB Summit 2019, our second community conference taking place November 18 in San Diego, California. Register today to join the FoundationDB community in person! What to expect This year, we’ve organized FoundationDB Summit into a two-track event. The conference will kick off with opening remarks to all attendees. Then the morning will spit into two groups: Case Studies will showcase how adopters have been able to create solutions that fit their needs using FoundationDB

will showcase how adopters have been able to create solutions that fit their needs using FoundationDB FDB Internals will focus on new development projects on the core FoundationDB project There will then be a break for lunch and few lightning talks followed by the afternoon sessions: Real World FDB will include talks on the experiences people have had with FoundationDB in real deployments

will include talks on the experiences people have had with FoundationDB in real deployments Building Layers will discuss work under way on various layers sitting on top of the core key-value store The conference will then conclude with everyone coming together for a few closing talks. The full schedule is available on the summit website. Who should attend FoundationDB Summit is a technical conference that will bring together early adopters, core developers, and other community members. If you’re interested in learning more about the project, we encourage you to join us! FoundationDB Summit is also part of the KubeCon + CloudNativeCon Community Events Day, and it will be in the same San Diego Convention Center as the other KubeCon events. Join the community FoundationDB Summit is a great opportunity to join the community, learn from and meet others, and get involved. We believe that a strong open source project requires a vibrant community, and we hope you’ll join us in San Digeo to help grow that for FoundationDB. Registration is open! Before the conference, if you’re looking to get involved we also encourage your participation on the community forums, and welcome contributions. We look forward to seeing you at FoundationDB Summit in San Diego!

FoundationDB Summit 2019 We’re happy to announce that this year’s FoundationDB Summit will again be hosted in coordination with KubeCon+CloudNativeCon in San Diego, CA on November 18, 2019. We’re looking for great speakers, and the call for proposals is now open! We welcome talks from existing and new community members, of varying backgrounds and experience with FoundationDB and public speaking. The CFP deadline is Friday, August 9. Registration for attending the FDB Summit will open soon!

FoundationDB 6.1.8 Released FoundationDB 6.1.8 is now officially available! The latest FoundationDB release can be downloaded and installed as binaries from the downloads page (available for macOS, Windows, Linux), or as source from our GitHub repository. If you're already running FDB, also see our upgrade instructions. This release removes a few of the caveats that came with cross-region replication when it was introduced in 6.0: Improved write throughput during failures : One of the primary motivations for cross-region replication is to keep the database available even when all machines in a region are offline. The 6.1 release dramatically increases the performance of the system during region failures, allowing clusters to keep up with their existing workloads.

: One of the primary motivations for cross-region replication is to keep the database available even when all machines in a region are offline. The 6.1 release dramatically increases the performance of the system during region failures, allowing clusters to keep up with their existing workloads. Robust forced recoveries: If all the transaction logs are lost when a region fails, the system will wait for those transaction logs to come back before recovering to ensure the database remains consistent. However, in the worst case that those transaction logs are lost permanently, the 6.1 release adds a way to force the system to recover to the other region. Some other highlights from this release include: Batch priority transactions : Added the ability to execute transactions at batch priority. These transactions do not execute if the cluster is too busy to handle them. This feature makes it safer and easier to bulk insert data into a production cluster.

: Added the ability to execute transactions at batch priority. These transactions do not execute if the cluster is too busy to handle them. This feature makes it safer and easier to bulk insert data into a production cluster. Efficient consistent caching: Added a special key whose value is sent to clients at the beginning of every transaction. Clients can know that locally cached metadata is still valid as long as the value of this key has not changed. A full list of features, fixes, and other changes are documented in our release notes.

Announcing The FoundationDB Record Layer Today, we are excited to announce the open source release of the FoundationDB Record Layer! From its inception, FoundationDB was designed as a highly scalable key-value store with a simple API. Layers extend the functionality of the database by adding features and data models to allow new storage access patterns. Today we are releasing the FoundationDB Record Layer, which provides relational database semantics on top of FoundationDB. This layer features schema management, indexing facilities, and a rich set of query capabilities. The Record Layer is used in production at Apple to support applications and services for hundreds of millions of users. A record-oriented database on top of FoundationDB The Record Layer stores structured data, just like a relational database. Databases managed by the Record Layer support records with fields and types, an evolving schema, complex primary and secondary indexes, and declarative query execution. The Record Layer also includes features not typically found in a traditional relational database, such as support for complex nested data types, indexes on the commit-time of records, and indexes and queries that span different types of records. Built on top of FoundationDB, the Record Layer inherits FoundationDB's strong ACID semantics, reliability, and performance in a distributed setting. The Record Layer also uses FoundationDB's transactional semantics to provide features similar to a traditional relational database, but in a distributed setting. For example, the Record Layer's secondary indexes are maintained transactionally, so they're always up-to-date with the latest changes to the data. Transactions reduce the number of bugs in application code and greatly simplify application development. Built for Scale The Record Layer is built for a massive scale, allowing millions of discrete database instances to be managed within a single FoundationDB cluster. Its design and core feature set were built to scale to many millions of concurrent users and a diverse ecosystem of client applications each with unique data models and query access patterns. To make operations simple, the Record Layer is stateless, so scaling up is as simple as launching more instances. It's also built from the ground up for massive multi-tenancy, encapsulating and isolating all of a tenant's state. The Record Layer can also tightly constrain and balance resource consumption across users in a predictable fashion, even in the face of a wide variety of workloads and activity. Together, these properties enable the Record Layer to scale elastically as the demands of its clients grow. Together, the Record Layer and FoundationDB form the backbone of Apple's CloudKit. We wrote a paper describing how we built the Record Layer to run at massive scale and how CloudKit uses it. Today, you can read the preprint to learn more. Highlights There's a lot more to say about the Record Layer. At a very high level, the Record Layer: Represents records as Protocol Buffer messages, providing industry-standard serialization and schema evolution. Features such as nested and repeated fields have first-class support.

Supports transactional secondary indexing that takes full advantage of the Protocol Buffer data model. We have a variety of advanced index types, including aggregate indexes such as grouped counts, full text indexes, ordinal rank indexes, and extensible functional indexes. Where possible, our implementations leverage advanced FoundationDB features, such as atomic mutations.

Includes a declarative query API for retrieving data and a query planner for turning those queries into concrete database operations.

Operates in a completely stateless manner. Instantiating a logical database and performing an operation can be done in milliseconds. Resource constraints allow any given operation or query to be limited. Continuations allow control to be handed back to the client to allow them to iteratively make progress in concert with all other clients.

Provides a large number of deep extension points. For example, users can build custom index maintainers and query planning features to seamlessly “plug-in” new index types. Similarly, the layer's serialization API supports client-defined encryption and compression algorithms. All of this leverages the full breadth and power of FoundationDB functionality, including strong ACID semantics and advanced features such as controllable isolation levels. Get Started The getting started guide will walk you through creating a simple application that uses the Record Layer. If you'd like to dig deeper, we wrote an overview that goes into greater detail. We encourage your participation in a new forum section for the Record Layer where you can ask discuss development, ask questions, and get involved in the open source project. We're excited to see what the community will build on top of the Record Layer. More information can be found at: Project documentation

Source on GitHub

Preprint paper

Announcing FoundationDB Document Layer Today we’re open sourcing the FoundationDB Document Layer, a document-oriented database that extends the core functionality of the FoundationDB key-value store. The FoundationDB Document Layer provides the ease-of-use of a document database in the form of the familiar MongoDB® API. The Document Layer is a stateless server, backed by the scalable and transactional features of FoundationDB. Released under an Apache v2 license, the Document Layer project is hosted on GitHub and binary downloads are available for macOS and Linux. See the project documentation to get started, and we encourage your participation on the new forum section for the Document Layer. MongoDB® API Compatible For developers looking to use FoundationDB, the Document Layer has a familiar API, and is compatible with the MongoDB® protocol. In fact, simple applications leveraging MongoDB® can have a lift-and-shift migration to the Document Layer. If you want to connect your application to the Document Layer, you can use any existing MongoDB® client — making it easy to get started. A discussion of the features supported by Document Layer can be found in the project's documentation. Scalable Document Storage with FoundationDB By extending FoundationDB, the Document Layer inherits key qualities of the core project — including years of testing, support for ACID transactions, and great performance. Using FoundationDB as a base for this stateless layer lets the Document Layer innovate in a number of notable ways. No write locks: write operations, even on a single document, do not take locks. Transactions in FoundationDB are optimistic (lockless) and, by storing a document as multiple key-value pairs, the Document Layer supports many parallel writers. In some cases, operations may have to retry due to conflicts; however, non-conflicting operations, including metadata operations, can proceed in parallel.

No sharding: The Document Layer does not rely on a fixed shard key to distribute data. Instead, all data partitioning and rebalancing is managed automatically by the key-value store. This design, directly inherited from FoundationDB, provides robust horizontal scalability while avoiding client-level complexity.

Easy scaling: Each running instance of the Document Layer is a stateless application, configured only with the FoundationDB cluster to which it stores data. This stateless design means that many instances of Document Layer can be put behind a load balancer, all able to handle queries from any client and for any document.

Safe defaults: By default, write operations on the Document Layer execute with full isolation and atomicity. This consistency makes it easier to correctly implement applications that handle more than one simultaneous request. Indexes are always consistent with document data. Writes are fully consistent all the time irrespective of client selected options and, similarly, reads always see strong causal consistency. Please see the documentation for more info about the architecture of the Document Layer. The Future is Layers When we first released FoundationDB we wrote in the original announcement: The vision of FoundationDB is to start with a simple, powerful core and extend it through the addition of “layers”. The key-value store, which is open sourced today, is the core, focused on incorporating only features that aren’t possible to write in layers. Layers extend that core by adding features to model specific types of data and handle their access patterns. The FoundationDB key-value store is powerful but, as noted above, its features remain narrowly scoped to distributed transactions and stateful storage. The Document Layer takes the key-value store's focused API and uses it to model a much more complex style of data storage. Because the Document Layer exposes a new, general-purpose API we think of this as an extension to FoundationDB. The key-value store of FoundationDB lacks a number of features you would expect of a full database system. For instance there are no indexes, no data typing, and no query engine. The Document Layer extends FoundationDB to add a number of such concepts. Take index maintenance: key-value store transactions are used to simultaneously update both a document's field and an index value that points back at the field. This is a common pattern for how simple indexes can be built. As an extension to FoundationDB, Document Layer makes this pattern available to clients in a robust and reusable manner. We could not be more excited to have the Document Layer as a community project! We're excited to see where we can take this together. Community collaboration does not need to end with the document model either — many other layers could be generally useful, written either as a library or a network server. The Document Layer should serve as a template for future layers both in design and in utility. Disclaimer: MongoDB is a registered trademark of MongoDB, Inc..

FoundationDB 6.0.15 Released FoundationDB 6.0.15, our first major release since open sourcing FoundationDB in April, is now officially available! The latest FoundationDB release can be downloaded and installed as binaries from the downloads page (available for macOS, Windows, Linux), or as source from our GitHub repository. If you're already running FDB, also see our upgrade instructions. The FoundationDB 6.0 series features an architectural shift in the approach to cross-region replication, improving how FoundationDB can be managed as a production system. Among the key improvements: New multi-region support within single clusters

TLS improved for operational flexibility

Faster failure recovery in a number of situations A full list of features, fixes, and other changes are documented in our release notes. New multi-region support and seamless failover FoundationDB 6.0 introduces native multi-region support to dramatically increase your database's global availability. Seamless failover between regions is now possible, allowing your cluster to survive the near-simultaneous loss of an entire region with no service interruption. These features can be deployed so clients experience low-latency, single-region writes. FoundationDB now has the flexibility to run in either multiple or single regions, offering greater control over how failover scenarios are managed. If you choose a multi-region configuration, one region will operate as the write authority by default. If an availability zone were to experience an outage, FoundationDB would automatically transfer authority to the other region without any data loss. Alternatively, if you choose to run FoundationDB in a single region, a FoundationDB cluster can now be configured to intelligently make use of multiple AZs for tolerance of an instant loss of an AZ. These features were made possible by adding several new concepts to FoundationDB: Region locality: When FoundationDB stores data to a disk, whether on a storage server or transaction log, it now has awareness of the region the disk is in. Locality information makes new features within FoundationDB possible, including the ability to prioritize process responsibilities so that region-local operations are always preferred.

When FoundationDB stores data to a disk, whether on a storage server or transaction log, it now has awareness of the region the disk is in. Locality information makes new features within FoundationDB possible, including the ability to prioritize process responsibilities so that region-local operations are always preferred. Bandwidth-conserving replication: New functionality conserves bandwidth when replicating data over expensive, high-latency intra-region links. This replication strategy sends a single copy of data between regions and then redistributes the data to replicas on the remote side.

New functionality conserves bandwidth when replicating data over expensive, high-latency intra-region links. This replication strategy sends a single copy of data between regions and then redistributes the data to replicas on the remote side. Region failure modes: The addition of multi-region support led us to rethink our response to failures across the codebase. FoundationDB has always been robust against machine failures and assumed these failures to be permanent. The response to a failure was to immediately copy data onto other machines. Region or AZ failures are, however, temporary and expensive to recover from, since copying the entire database's contents over the WAN is expensive. These architectural changes open up a host of possibilities for future replication configurations and datacenter layouts. To learn more about configuring region settings, please check out the updated documentation and discuss these changes further on the FoundationDB forums. Thanks to our contributors Thanks to those who contributed to the 6.0 series: A.J. Beamon, Alec Grieser, Alex Miller, Alvin Moore, Balachandar Namasivayam, Bhaskar Muppana, Caleb Spare, Clement Pang, Evan Tschannen, John Brownlee, Justin Lowery, Richard Low, Steve Atherton, Jason Moody, and Ryan Worl. Community feedback and contributions are welcome! If you’d like to contribute back to the project please see our contributors guide. Join the community at FoundationDB Summit The FoundationDB community is hosting its first-ever conference, FoundationDB Summit, on December 10 in Seattle, WA. There's still time to register and attend, and we hope to see you there!

Official Swift bindings for FoundationDB Today we are announcing the official release of FoundationDB bindings for Swift. Now you can write native Swift code to interact with directly with the FoundationDB key-value store. The official FoundationDB Swift bindings are available via GitHub: https://github.com/FoundationDB/fdb-swift-bindings, and we've published API documentation to get started. Swift is the latest addition to several officially-supported language bindings, including Java, Python, Ruby, Go, C, and Flow. Evolving with the community Like other official bindings, we’re committed to maintaining the compatibility of the Swift bindings with future releases. This initial release supports a subset of the functionality of other clients, with future work planned including a directory layer. Our intention is to evolve and test the bindings with community feedback, and we welcome your contributions.

Announcing FoundationDB Summit Today we're announcing our first-ever FoundationDB community conference: FoundationDB Summit. FoundationDB Summit will take place December 10 in Seattle at the Washington Convention Center, and is organized in partnership with the Linux Foundation and the Cloud Native Computing Foundation (CNCF). Participating in the summit We’re looking for great speakers, and the call for proposals is now open! We welcome talks from existing and new community members, of varying backgrounds and experience with FoundationDB and public speaking. The CFP deadline is Friday, September 21. Registration is open: Early-bird tickets: $99 until Friday, August 31

Standard tickets: $150 We’ve also established diversity scholarships for applicants from underrepresented and/or marginalized groups. A limited number of sponsorship opportunities for the event exist, and are a great way to give back to the community. An opportunity to learn from the community FoundationDB Summit is organized on a single track with plenty of time to meet and learn from early adopters, core developers, and other community members. We're also part of the KubeCon + CloudNativeCon Community Events Day, and will be in the same Washington Convention Center as thousands of other open source engineers that week. We believe FoundationDB can become the bedrock for the next generation of distributed databases, and to achieve that it’s imperative that we continue to grow and nurture the community around it. Whether you're an existing FoundationDB user or new to the project, we hope to see you at the event!

FoundationDB support added for HashiCorp Vault With its recent 0.10.4 release, HashiCorp Vault has added a FoundationDB physical backend. This feature adds stateful storage of secrets into of a distributed and highly-available backend. Additionally, this backend implements Vault's transactional interface, needed for some specialized features. The code powering this new Vault feature is contained in the original pull request #4900, along with configuration information in the Vault documentation. The adapter uses FoundationDB's Go bindings.