Announcing The FoundationDB Record Layer

Today, we are excited to announce the open source release of the FoundationDB Record Layer!

From its inception, FoundationDB was designed as a highly scalable key-value store with a simple API. Layers extend the functionality of the database by adding features and data models to allow new storage access patterns. Today we are releasing the FoundationDB Record Layer, which provides relational database semantics on top of FoundationDB. This layer features schema management, indexing facilities, and a rich set of query capabilities. The Record Layer is used in production at Apple to support applications and services for hundreds of millions of users.

A record-oriented database on top of FoundationDB

The Record Layer stores structured data, just like a relational database. Databases managed by the Record Layer support records with fields and types, an evolving schema, complex primary and secondary indexes, and declarative query execution. The Record Layer also includes features not typically found in a traditional relational database, such as support for complex nested data types, indexes on the commit-time of records, and indexes and queries that span different types of records.

Built on top of FoundationDB, the Record Layer inherits FoundationDB's strong ACID semantics, reliability, and performance in a distributed setting. The Record Layer also uses FoundationDB's transactional semantics to provide features similar to a traditional relational database, but in a distributed setting. For example, the Record Layer's secondary indexes are maintained transactionally, so they're always up-to-date with the latest changes to the data. Transactions reduce the number of bugs in application code and greatly simplify application development.

Built for Scale

The Record Layer is built for a massive scale, allowing millions of discrete database instances to be managed within a single FoundationDB cluster. Its design and core feature set were built to scale to many millions of concurrent users and a diverse ecosystem of client applications each with unique data models and query access patterns.

To make operations simple, the Record Layer is stateless, so scaling up is as simple as launching more instances. It's also built from the ground up for massive multi-tenancy, encapsulating and isolating all of a tenant's state. The Record Layer can also tightly constrain and balance resource consumption across users in a predictable fashion, even in the face of a wide variety of workloads and activity. Together, these properties enable the Record Layer to scale elastically as the demands of its clients grow.

Together, the Record Layer and FoundationDB form the backbone of Apple's CloudKit. We wrote a paper describing how we built the Record Layer to run at massive scale and how CloudKit uses it. Today, you can read the ~~preprint~~ SIGMOD'19 to learn more.

Highlights

There's a lot more to say about the Record Layer. At a very high level, the Record Layer:

• Represents records as Protocol Buffer messages, providing industry-standard serialization and schema evolution. Features such as nested and repeated fields have first-class support.
• Supports transactional secondary indexing that takes full advantage of the Protocol Buffer data model. We have a variety of advanced index types, including aggregate indexes such as grouped counts, full text indexes, ordinal rank indexes, and extensible functional indexes. Where possible, our implementations leverage advanced FoundationDB features, such as atomic mutations.
• Includes a declarative query API for retrieving data and a query planner for turning those queries into concrete database operations.
• Operates in a completely stateless manner. Instantiating a logical database and performing an operation can be done in milliseconds. Resource constraints allow any given operation or query to be limited. Continuations allow control to be handed back to the client to allow them to iteratively make progress in concert with all other clients.
• Provides a large number of deep extension points. For example, users can build custom index maintainers and query planning features to seamlessly “plug-in” new index types. Similarly, the layer's serialization API supports client-defined encryption and compression algorithms.

All of this leverages the full breadth and power of FoundationDB functionality, including strong ACID semantics and advanced features such as controllable isolation levels.

Get Started

The getting started guide will walk you through creating a simple application that uses the Record Layer. If you'd like to dig deeper, we wrote an overview that goes into greater detail.

We encourage your participation in a new forum section for the Record Layer where you can ask discuss development, ask questions, and get involved in the open source project. We're excited to see what the community will build on top of the Record Layer.

More information can be found at:

• Project documentation
• Source on GitHub
• SIGMOD'19 paper

Updated 2020/10/24: Link to SIGMOD'19 paper instead or preprint.