Announcing FoundationDB Summit, our community conference. December 10, 2018 in Seattle, WA. Registration open!

Announcing FoundationDB Document Layer

Published November 29, 2018

Today we’re open sourcing the FoundationDB Document Layer, a document-oriented database that extends the core functionality of the FoundationDB key-value store.

The FoundationDB Document Layer provides the ease-of-use of a document database in the form of the familiar MongoDB® API. The Document Layer is a stateless server, backed by the scalable and transactional features of FoundationDB.

Released under an Apache v2 license, the Document Layer project is hosted on GitHub and binary downloads are available for macOS and Linux. See the project documentation to get started, and we encourage your participation on the new forum section for the Document Layer.

MongoDB® API Compatible

For developers looking to use FoundationDB, the Document Layer has a familiar API, and is compatible with the MongoDB® protocol. In fact, simple applications leveraging MongoDB® can have a lift-and-shift migration to the Document Layer.

If you want to connect your application to the Document Layer, you can use any existing MongoDB® client — making it easy to get started. A discussion of the features supported by Document Layer can be found in the project's documentation.

Scalable Document Storage with FoundationDB

By extending FoundationDB, the Document Layer inherits key qualities of the core project — including years of testing, support for ACID transactions, and great performance. Using FoundationDB as a base for this stateless layer lets the Document Layer innovate in a number of notable ways.

  • No write locks: write operations, even on a single document, do not take locks. Transactions in FoundationDB are optimistic (lockless) and, by storing a document as multiple key-value pairs, the Document Layer supports many parallel writers. In some cases, operations may have to retry due to conflicts; however, non-conflicting operations, including metadata operations, can proceed in parallel.
  • No sharding: The Document Layer does not rely on a fixed shard key to distribute data. Instead, all data partitioning and rebalancing is managed automatically by the key-value store. This design, directly inherited from FoundationDB, provides robust horizontal scalability while avoiding client-level complexity.
  • Easy scaling: Each running instance of the Document Layer is a stateless application, configured only with the FoundationDB cluster to which it stores data. This stateless design means that many instances of Document Layer can be put behind a load balancer, all able to handle queries from any client and for any document.
  • Safe defaults: By default, write operations on the Document Layer execute with full isolation and atomicity. This consistency makes it easier to correctly implement applications that handle more than one simultaneous request. Indexes are always consistent with document data. Writes are fully consistent all the time irrespective of client selected options and, similarly, reads always see strong causal consistency.

Please see the documentation for more info about the architecture of the Document Layer.

The Future is Layers

When we first released FoundationDB we wrote in the original announcement:

The vision of FoundationDB is to start with a simple, powerful core and extend it through the addition of “layers”. The key-value store, which is open sourced today, is the core, focused on incorporating only features that aren’t possible to write in layers. Layers extend that core by adding features to model specific types of data and handle their access patterns.

The FoundationDB key-value store is powerful but, as noted above, its features remain narrowly scoped to distributed transactions and stateful storage. The Document Layer takes the key-value store's focused API and uses it to model a much more complex style of data storage. Because the Document Layer exposes a new, general-purpose API we think of this as an extension to FoundationDB.

The key-value store of FoundationDB lacks a number of features you would expect of a full database system. For instance there are no indexes, no data typing, and no query engine. The Document Layer extends FoundationDB to add a number of such concepts. Take index maintenance: key-value store transactions are used to simultaneously update both a document's field and an index value that points back at the field. This is a common pattern for how simple indexes can be built. As an extension to FoundationDB, Document Layer makes this pattern available to clients in a robust and reusable manner.

We could not be more excited to have the Document Layer as a community project! We're excited to see where we can take this together. Community collaboration does not need to end with the document model either — many other layers could be generally useful, written either as a library or a network server. The Document Layer should serve as a template for future layers both in design and in utility.

Disclaimer: MongoDB is a registered trademark of MongoDB, Inc..

FoundationDB 6.0.15 Released

Published November 19, 2018

FoundationDB 6.0.15, our first major release since open sourcing FoundationDB in April, is now officially available!

The latest FoundationDB release can be downloaded and installed as binaries from the downloads page (available for macOS, Windows, Linux), or as source from our GitHub repository. If you're already running FDB, also see our upgrade instructions.

The FoundationDB 6.0 series features an architectural shift in the approach to cross-region replication, improving how FoundationDB can be managed as a production system. Among the key improvements:

  • New multi-region support within single clusters
  • TLS improved for operational flexibility
  • Faster failure recovery in a number of situations

A full list of features, fixes, and other changes are documented in our release notes.

New multi-region support and seamless failover

FoundationDB 6.0 introduces native multi-region support to dramatically increase your database's global availability. Seamless failover between regions is now possible, allowing your cluster to survive the near-simultaneous loss of an entire region with no service interruption. These features can be deployed so clients experience low-latency, single-region writes.

FoundationDB now has the flexibility to run in either multiple or single regions, offering greater control over how failover scenarios are managed. If you choose a multi-region configuration, one region will operate as the write authority by default. If an availability zone were to experience an outage, FoundationDB would automatically transfer authority to the other region without any data loss. Alternatively, if you choose to run FoundationDB in a single region, a FoundationDB cluster can now be configured to intelligently make use of multiple AZs for tolerance of an instant loss of an AZ.

These features were made possible by adding several new concepts to FoundationDB:

  • Region locality: When FoundationDB stores data to a disk, whether on a storage server or transaction log, it now has awareness of the region the disk is in. Locality information makes new features within FoundationDB possible, including the ability to prioritize process responsibilities so that region-local operations are always preferred.
  • Bandwidth-conserving replication: New functionality conserves bandwidth when replicating data over expensive, high-latency intra-region links. This replication strategy sends a single copy of data between regions and then redistributes the data to replicas on the remote side.
  • Region failure modes: The addition of multi-region support led us to rethink our response to failures across the codebase. FoundationDB has always been robust against machine failures and assumed these failures to be permanent. The response to a failure was to immediately copy data onto other machines. Region or AZ failures are, however, temporary and expensive to recover from, since copying the entire database's contents over the WAN is expensive.

These architectural changes open up a host of possibilities for future replication configurations and datacenter layouts. To learn more about configuring region settings, please check out the updated documentation and discuss these changes further on the FoundationDB forums.

Thanks to our contributors

Thanks to those who contributed to the 6.0 series: A.J. Beamon, Alec Grieser, Alex Miller, Alvin Moore, Balachandar Namasivayam, Bhaskar Muppana, Caleb Spare, Clement Pang, Evan Tschannen, John Brownlee, Justin Lowery, Richard Low, Steve Atherton, Jason Moody, and Ryan Worl.

Community feedback and contributions are welcome! If you’d like to contribute back to the project please see our contributors guide.

Join the community at FoundationDB Summit

The FoundationDB community is hosting its first-ever conference, FoundationDB Summit, on December 10 in Seattle, WA. There's still time to register and attend, and we hope to see you there!

FoundationDB Summit Program Announced

Published October 25, 2018

We’re pleased to announce the program for FoundationDB Summit, our inaugural community conference taking place December 10 in Seattle, Washington.

Register today to join the FoundationDB community in person!

What to expect

We’ve organized FoundationDB Summit into a single-track event, with plenty of time to meet members of the community and learn from one another.

The conference will kickoff with an introduction to the project, share updates since it was open sourced in April, and offer a technical overview of FoundationDB.

The rest of the day will pick up the pace and take us on a tour of what’s happening in the community, focusing on three areas:

These are just samples of some of the talks that are planned. The event schedule is now online, and we encourage you to check out the full program which includes lightning talks.

Who should attend

FoundationDB Summit is a technical conference that will bring together early adopters, core developers, and other community members. If you’re interested in learning more about the project, we encourage you to join us!

We’ve also established diversity scholarships for applicants from underrepresented and/or marginalized groups. The deadline to apply for scholarships is next Friday, November 2.

FoundationDB Summit is also part of the KubeCon + CloudNativeCon Community Events Day, and will be in the same Washington Convention Center as thousands of other open source engineers that week.

Join the community

FoundationDB Summit is a great opportunity to join the community, learn from and meet others, and get involved. We believe that a strong open source project requires a vibrant community, and we hope you’ll join us in Seattle to help grow that for FoundationDB. Registration is open!

Before the conference, if you’re looking to get involved we also encourage your participation on the community forums, and welcome contributions.

We look forward to seeing you at FoundationDB Summit in Seattle!

Official Swift bindings for FoundationDB

Published October 4, 2018

Today we are announcing the official release of FoundationDB bindings for Swift. Now you can write native Swift code to interact with directly with the FoundationDB key-value store.

The official FoundationDB Swift bindings are available via GitHub: https://github.com/FoundationDB/fdb-swift-bindings, and we've published API documentation to get started.

Swift is the latest addition to several officially-supported language bindings, including Java, Python, Ruby, Go, C, and Flow.

Evolving with the community

Like other official bindings, we’re committed to maintaining the compatibility of the Swift bindings with future releases.

This initial release supports a subset of the functionality of other clients, with future work planned including a directory layer. Our intention is to evolve and test the bindings with community feedback, and we welcome your contributions.

Announcing FoundationDB Summit

Published August 22, 2018

Today we're announcing our first-ever FoundationDB community conference: FoundationDB Summit.

FoundationDB Summit will take place December 10 in Seattle at the Washington Convention Center, and is organized in partnership with the Linux Foundation and the Cloud Native Computing Foundation (CNCF).

Participating in the summit

We’re looking for great speakers, and the call for proposals is now open! We welcome talks from existing and new community members, of varying backgrounds and experience with FoundationDB and public speaking. The CFP deadline is Friday, September 21.

Registration is open:

  • Early-bird tickets: $99 until Friday, August 31
  • Standard tickets: $150

We’ve also established diversity scholarships for applicants from underrepresented and/or marginalized groups.

A limited number of sponsorship opportunities for the event exist, and are a great way to give back to the community.

An opportunity to learn from the community

FoundationDB Summit is organized on a single track with plenty of time to meet and learn from early adopters, core developers, and other community members. We're also part of the KubeCon + CloudNativeCon Community Events Day, and will be in the same Washington Convention Center as thousands of other open source engineers that week.

We believe FoundationDB can become the bedrock for the next generation of distributed databases, and to achieve that it’s imperative that we continue to grow and nurture the community around it. Whether you're an existing FoundationDB user or new to the project, we hope to see you at the event!

FoundationDB support added for HashiCorp Vault

Published July 31, 2018

With its recent 0.10.4 release, HashiCorp Vault has added a FoundationDB physical backend. This feature adds stateful storage of secrets into of a distributed and highly-available backend. Additionally, this backend implements Vault's transactional interface, needed for some specialized features.

The code powering this new Vault feature is contained in the original pull request #4900, along with configuration information in the Vault documentation. The adapter uses FoundationDB's Go bindings.

FoundationDB 5.2.5 Released

Published June 26, 2018

FoundationDB 5.2.5 has officially been released! This patch release is the part of the 5.2 series, and underwent significant testing. We’re pleased to now make it available on our website.

The latest FoundationDB release can be downloaded and installed as binaries from our downloads page (available for macOS, Windows, Linux), or as source from our GitHub repository. If you're already running FDB, also see our upgrade instructions.

Community feedback and contributions are welcome! The FoundationDB forums have both user and development sections, and if you’d like to contribute back to the project please see our contributors guide.

5.2.5 release highlights

Several changes stand out within this release, including:

  • A TLS plugin implementation has been merged into the master branch. (PR #343)
  • Improved backup task prioritization. (PR #71)
  • Backup and DR share a single mutation log when both are being used on the same cluster. Ongoing backups will be aborted when upgrading to 5.2. (PR #3)
  • Added the APPEND_IF_FITS atomic operation. (PR #22)
  • Added status metrics for read bytes per second and read keys per second. (PR #303)

A full list of features, fixes, and other changes are documented within our release notes.

FoundationDB community highlights, two weeks in

Published May 3, 2018

In two weeks since open sourcing FoundationDB, we've had an outpouring of interest.

Over 100 new topics have been opened on the community forums, 7400+ developers have starred the project on GitHub, and we've seen numerous code contributions from developers who are exploring FoundationDB for the first time. Thank you to all who have installed FoundationDB, contributed, and engaged in our community.

Coinciding with the release announcement, several long-time adopters have also blogged about their experiences with the project, including Wavefront and Snowflake:

We believe in, and want to build a strong community to help support and develop the project. We're thrilled by this early participation.

Early activity in the ecosystem

FoundationDB is a multi-model system, enabling different layers to extend the core key-value store for different uses. One of the things we're most excited to see is development outside of the core, with the community extending the project in new ways.

Since the initial open source release, we've seen several community-driven projects emerge. Here are a few worth checking out:

Layers:

  • JanusGraph is a distributed graph database. The development of a storage adapter enables it to be used with FoundationDB. See the forum post for more info.
  • Two projects to implement a network block device, one in Python, the other in Java.

Language bindings:

This is only the beginning, and we can't wait to see what you build with FoundationDB! Be sure to share your experience on the community forums.

Thank you

Lastly, thank you to all who have contributed to the FoundationDB core and already had your code merged into the master branch the past two weeks: A.J. Beamon, Alec Grieser, Alex Miller, Alexander Lagerström, Alvin Moore, Amanda Aizuss, Ben Halverson, Bruce Mitchener, Chr1st0ph, Dave Lester, Dennis Schafroth, Douglas Daniels, Evan Tschannen, Hiroshi Saito, Iuri Sitinschi, John Brownlee, Martin Junker, Matias Insaurralde, Semih Tok, Steve Malmskog, Umar Farouk Umar, Vince Polsinelli, Yichi Chiang, naru-jpn, tracebundy, and xtreak.

Contributions are welcome! For more information on getting involved, please see our contributors guide.

FoundationDB is Open Source

Published April 19, 2018

The next chapter

Starting today, FoundationDB starts its next chapter as an open source project!

FoundationDB is a distributed datastore, designed from the ground up to be deployed on clusters of commodity hardware. These clusters scale well as you add machines, automatically heal from hardware failures, and have a simple API. The key-value store supports fully global, cross-row ACID transactions. That's the highest level of data consistency possible. What does this mean for you? Strong consistency makes your application code simpler, your data models more efficient, and your failure modes less surprising.

The great thing is that FoundationDB is already well-established — it's actively developed and has years of production use. We intend to drive FoundationDB forward as a community project and we welcome your participation.

A powerful abstraction

We believe FoundationDB can become the foundation of the next generation of distributed databases. Since its beginnings in 2010 as a startup, the world of databases has increasingly aligned with FoundationDB to favor data consistency.

The vision of FoundationDB is to start with a simple, powerful core and extend it through the addition of “layers”. The key-value store, which is open sourced today, is the core, focused on incorporating only features that aren’t possible to write in layers. Layers extend that core by adding features to model specific types of data and handle their access patterns.

The fundamental architecture of FoundationDB, including its use of layers, promotes the best practices of scalable and manageable systems. By running multiple layers on a single cluster (for example a document store layer and a graph layer), you can match your specific applications to the best data model. Running less infrastructure reduces your organization's operational and technical overhead.

By open sourcing the FoundationDB core, we expect the quantity and variety of layers to develop rapidly. When we think about the FoundationDB community, we approach it both in terms of the core itself and the ecosystem of layers that it enables.

Building an Open Community

By open sourcing FoundationDB, our goal is to build an open community. All major development will be done in the open. We’ve outlined a design document process to ensure that this work is done transparently and with community input. We’ve taken early steps to outline project governance to provide a basic structure that will enable members of the community who actively contribute to have a greater voice in the project decision-making.

We also want FoundationDB to be a healthy and responsive community. To that end, we’ve adopted a code of conduct based on the Contributor Covenant to outline the behaviors we encourage and those we disallow.

We’d love your participation. Here are several ways you can get involved:

  • Ask questions on the FoundationDB community forums: forums.foundationdb.org. We have categories for user-related questions (how do I use X) as well as development questions (I am digging into the FoundationDB core and want to change Y). Say hello!
  • Help improve the software by reporting bugs through GitHub issues.
  • Make contributions to the core software and documentation (please see the project’s contribution guide).

Get Started

The source for FoundationDB is available at github.com/apple/foundationdb.

Please see the Getting Started guide for the basics of how to install, use, and develop against FoundationDB. Binary installers are available for macOS, Windows, and Linux at www.foundationdb.org/download/.