Previously, we wrote our experiences while designing our system to tolerate these kinds of disastrous scenarios: Crossing the River — An Adventure of Cross-Region Replication with DynamoDB.

Degraded performance or losing data during a disaster case is unacceptable for a database service.

Until now, we implemented our custom solution for replicating DynamoDB data across multiple regions and monitoring the system. Custom solutions may be error-prone and hard to maintain. Luckily, AWS announced DynamoDB Global Table at AWS re:Invent 2017. In this post, we will share our experiences of going deep into this feature.

What is a Global Table?

Global Tables are multi-region, multi-master DynamoDB tables supporting fast local performance of globally distributed apps. While regular DynamoDB tables are replicated between multiple availability zones by default, this feature assures higher level of availability and durability by replicating our tables across regions. Global Tables help us focus on our business logic rather than struggling with replicating, monitoring and resolving conflicts. The best part is we don’t need to change our existing code or system to switch to Global Tables from regular DynamoDB tables. Unfortunately, things get a bit tricky when we want to migrate our existing tables into Global Tables. We will get into details of this later in this post.

Global Table at a glance

Multi-region, multi-master DynamoDB

If one of the AWS regions were to become temporarily unavailable, our app’s users could still access the same table in the other regions.

If one of the AWS regions were to become temporarily unavailable, our app’s users could still access the same table in the other regions. Asynchronous replication of data between the regions via DynamoDB Streams

No upfront cost or commitments, pay for;

DB storage we use — doubles the cost as we store our data in two regions.

Data transfer between regions while replicating.

DB storage we use — doubles the cost as we store our data in two regions. Data transfer between regions while replicating. Replicated within seconds

A newly-written item is “usually” propagated to all replica tables within seconds.

A newly-written item is “usually” propagated to all replica tables within seconds. No update conflicts

last writer wins based on update time values in records.

last writer wins based on update time values in records. Eventual consistency for replicated tables

We should prefer the same region for strongly consistent reads.

We should prefer the same region for strongly consistent reads. New fields are added or updated for each item during replication. These fields should not be modified.

aws:rep:updateregion

The region that owns the data (i.e., us-west-2)

aws:rep:updatetime

The timestamp that the item is updated (i.e., 1514906551.627001)

The region that owns the data (i.e., us-west-2) The timestamp that the item is updated (i.e., 1514906551.627001) After replication is completed, all fields in a record are exactly the same.

Sample records replicated in a global table

DynamoDB keeps track of any writes that have been performed but have not been yet propagated to replicas.

It resumes propagating any pending writes to other regions as well as to current region from other regions.

There may be some conflicts that should be handled by ourselves (we call it logical conflicts and will explain it in the following parts).

There may be race conditions between applications in different regions that update the very same record.

Keeping track of pending updates, or race conditions are outsourced

There is not any built-in mechanism that informs us in case of a regional degradation. Thus we should apply custom business logic to determine when to redirect requests to other regions (monitoring metrics may be useful to do so).

In order to achieve multi-region fault tolerance, we need to modify our application to failover to another region (we may use a second DynamoDB client and switch to another client to direct traffic to another region).

Constraints

All tables should be owned by a single AWS account.

Currently available regions are N.Virginia, Ohio, Oregon, Ireland, and Frankfurt.

Tables should have the same name and key configuration (hash key, sort key).

Tables should be empty.

Streams should be enabled (for new and old images).

Tables do not support partial replication of only some of the items or strongly consistent reads from different regions.

Best Practices

Global Tables in all regions should share a consistent set of:

Auto Scaling

TTL

Local Secondary Index

Global Secondary Index

Provisioned throughput settings (The read capacity units (RCUs) and write capacity units (WCUs) for each global secondary index should likewise be identical on every replica)

IAM policies

Auto Scaling TTL Local Secondary Index Global Secondary Index Provisioned throughput settings (The read capacity units (RCUs) and write capacity units (WCUs) for each global secondary index should likewise be identical on every replica) IAM policies RCUs/WCUs and partition counts should be same for all regions to provide consistency; otherwise, one replica may fall further behind.

Using auto-scale with the same configuration for both regions is recommended; otherwise, we should manage capacities for all regions manually.

Monitoring Global Tables

We can use Amazon CloudWatch to monitor the behavior and performance of a Global Table. Since there is not any built-in mechanism that informs us in case of a regional degradation, monitoring is crucial to realize problems early and make the failover decision automatically.

Amazon DynamoDB publishes two metrics for each replica in the global table:

ReplicationLatency: elapsed time of propagating

elapsed time of propagating PendingReplicationCount: number of items written to one replica but not propagated to other regions

We should monitor replication processes constantly and create alarms based on these metrics.

If the numbers are high enough for us to determine that the region is experiencing problems, we can temporarily redirect our application’s read and write activity to a different AWS region.

ReplicationLatency and PendingReplicationCount

Oregon and Ohio Write Capacities respectively

Logical Conflicts

Global Tables support conflict resolution by using last writer wins approach. But still, there is a chance of experiencing a logical conflict. A logical conflict is the denial of a write request by AWS just because the timestamp is less than the latest timestamp of the record. This is an unexpected situation that may occur after switching traffic to another region.

To give a specific example, let’s assume that we have a count field in a table where t symbolizes the timestamp;

t=0 :: count=3 in both Oregon and Ohio regions

t=1 :: there is a connection issue between the regions

t=2 :: App-Oregon updates count to 5

t=3 :: App-Ohio updates count to 8

Note that because of the network issue, modifications couldn’t be propagated.

Note that because of the network issue, modifications couldn’t be propagated. t=4 :: count=5 in Oregon whereas count=8 in Ohio at the same time

t=5 :: network issue resolved

t=6 :: count will be 8 in both regions since t=3 > t=2 (last writer wins)

We should be notified of a conflict that AWS solved by comparing the timestamps because this may ruin data integrity and data integrity could be crucial for some systems. Unfortunately, there is no built-in solution or metric supporting such notifcation for now.

Race Conditions

Since Global Table does not support partial updates, data integrity may be broken. If we are running multiple applications in different regions which are using a Global Table, we should consider race conditions.

Let’s assume that we have two applications: App-Oregon and App-Ohio which are using Lecture table (a Global Table). App-Oregon updates Tutor from John to Samantha, and App-Ohio updates lecture time 2PM to 4PM at the very same time only with a nanosecond-level difference. In this case, both modification events will be in-transit to other region at the same time. Because Global Table resolves conflicts by using last writer wins approach, one of the modifications will be accepted, and the other one will be rejected. Therefore, data integrity will probably be broken, and the record will be as “John, 4PM” or “Samantha, 2PM” but not “Samantha, 4PM” which is the expected result eventually.

How to Convert Existing Tables into Global Tables

As mentioned before, there is not any solution supported by AWS at the moment. Here is the solution that we suggest:

If we can stop the instances or database traffic for a while,

We can export our data to S3 and import to new tables using AWS Data Pipeline.

Export data from DynamoDB

Import data into DynamoDB We can enable streams and migrate all data using a Lambda function. Unfortunately, we can’t use Backup and Restore Service for now since we can’t create a global table while restoring.

For the systems that we can’t stop incoming traffic, we need to implement our custom solution (blue/green approach):

Create a new global table (old table: Customer > new table: CustomerV2). Modify our application code to:

use the old table for writes

use the new table for reads (use old table if not found in new table) Enable stream and attach a Lambda function that migrates all data to the new table. After all data is migrated, update application logic to use the new table for both reads and writes. Delete the old table.

For more information about Global Tables, you can also check the following AWS resources;

Interested in more?