With both the F1 and Spanner papers out its now possible to understand their interplay a bit holistically. So lets start by revisiting the key goals of both systems.

Key Goals of F1’s design

Spanner’s objectives

At a logical level F1 has a data model that is similar to an RDBMS. Additionally tables in F1 can be organized into a hierarchy.

A row corresponding to the root table in the hierarchy is called the root row.

Rows of child tables related to the root row are stored in one single Spanner directory.

Client applications declare the hierarchies in database schemas via the INTERLEAVE IN declarations.

Each row in a directory table with key K, together with all of the rows in descendant tables that start with K in lexicographic order, forms a directory.

Physically, each child table is clustered with and interleaved within the rows from its parent table.

The paper goes on to highlight some of the benefits of having a hierarchical schema for both reads and writes. However hierarchical modeling is not mandatory in F1. A flat schema is also possible.

Indexes in F1 are transactional and fully consistent. Indexes are stored as separate tables in Spanner, keyed by a concatenation of the index key and the indexed table’s primary key.

It uses two types of physical storage layouts – Local and Global

Query Processing in F1

Its not very surprising to observe that query processing in F1 looks remarkably similar to what we find in the more recent SQL-on-Hadoop solutions such as Cloudera’s Impala, Apache Drill and predecessor shared nothing parallel databases.

Lifecycle of a query

Each query has a query coordinator node. Its the node that receives the SQL query request.

The coordinator plans the query for execution, receives results from the penultimate execution nodes, performs any final aggregation, sorting, or filtering, and then streams the results back to the client,

Given that data is arbitrarily partitioned the query planner determines the extent of parallelism needed to minimize the processing time for a query.

Based on the data required to be processed and scope for parallelism the planner/optimizer may even choose to repartition the qualifying data Dealing with network latencies

F1’s main data store is Spanner, which is a remote data source. F1 SQL can also access other remote data sources whose accesses involve highly variable network latency.

The issues associated with remote data access (which are issues due to network latency) are mitigated through the use of batching and pipelining across various stages of the query life cycle. Also the query operators are designed to stream as much data as possible to the subsequent stages of the processing pipeline.

And finally….

The F1 system has been managing all AdWords advertising campaign data in production since early 2012. AdWords is a vast and diverse ecosystem including 100s of applications and 1000s of users, all sharing the same database. This database is over 100 TB, serves up to hundreds of thousands of requests per second, and runs SQL queries that scan tens of trillions of data rows per day. Availability reaches five nines, even in the presence of unplanned outages, and observable latency on our web applications has not increased compared to the old MySQL system.