Are We Consistent Yet?

Observed and documented eventual consistency of object stores, e.g., Amazon S3, OpenStack Swift.

What is eventual consistency?

Traditional systems provide strong consistency, where clients can immediately view updates. Some distributed systems relax their consistency model to allow greater availability or better performance. Eventual consistency manifests itself to clients as stale views of data.

Observed consistency

We ran a test in which we wrote (i.e., create, update, delete) an object and then attempted to read the object. Across many trials, we count the number of times the object was not immediately available. When the object is not immediately found, it's an occurrence of observed eventual consistency.

Observed instances of eventual consistency with a number of operations during tests around 17 December 2014 with a 1-byte object size:

Provider read after create read after delete read after overwrite list after create list after delete number of operations Amazon S3 (us-standard) * * * 16 * 100,000 Amazon S3 (us-standard†) 16 8 12 8 3 1,000 Amazon S3 (us-west) - 1 1 * * 100,000 Ceph (DreamObjects) - - - - - 1,000 Google Cloud Storage - - - 2 2 1,000 Microsoft Azure Storage - - - - - 1,000 OpenStack Swift (Rackspace) - 17 3 30 20 1,000

Legend:

- zero observed instances

* zero observed instances but expect non-zero

† writing to N. Virginia and reading from Pacific Northwest

Documented consistency

Amazon S3 - buckets in the US Standard region provide eventual consistency. Buckets in all other regions provide read-after-write consistency for PUTs of new objects and eventual consistency for overwrite PUTs and DELETEs.

Ceph - provides well-defined safety semantics and strong consistency guarantees.

Google Cloud Storage - provides strong global consistency for all read-after-write, read-after-update, and read-after-delete operations, including both data and metadata. Bucket and object listing are also strongly consistent.

Microsoft Azure Storage - was designed to embrace a strong consistency model which guarantees that when the Storage service commits a data insert or update operation all further accesses to that data will see the latest update.

OpenStack Swift - For example, suppose a container server is under load and a new object is put in to the system. The object will be immediately available for reads as soon as the proxy server responds to the client with success. However, the container server did not update the object listing, and so the update would be queued for a later update. Container listings, therefore, may not immediately contain the object. Additional reference.

References