TL;DR: We moved our system to a different cloud provider and copied the MongoDB replica set database between two different Kubernetes clusters with zero downtime. This is how we did it.

Background

There might come a time in a SaaS company’s lifetime that it will decide it’s no longer happy with the infrastructure of the cloud provider it uses and would like to change providers. Assuming the system is designed properly and easy to deploy, this should not be too terrifying. However, in order to maintain a continuous service to its customers, the data needs to be migrated as well. And that, depending on the data volume, is no easy feat.

Since we constantly process data and can’t afford a long downtime we decided to keep the databases between the old and the new clusters in sync while deploying the system on the new cluster. This had several advantages over copying a snapshot of the data. First, this way we can avoid replaying a delta of the data changed during the migration window. Second, in case of any major issues discovered after the move, it would be easy to roll back to the old provider without losing any data.

Land Surveying

In order to connect MongoDB servers on separate Kubernetes clusters to a single replica set, we need to make sure that every member can access all other members directly by DNS name. First of all, we created the new members in the new k8s cluster, three MongoDB pods managed by a stateful set and a headless service with a different name then the service in the old cluster.

Now we have six instances, let’s call them:

mongo-0.mongo-old

mongo-1.mongo-old

mongo-2.mongo-old

mongo-0.mongo-new

mongo-1.mongo-new

mongo-2.mongo-new

The first three exist in the old k8s cluster and are configured in a replica set (pre existing before the migration). The last three in the new cluster and have not been configured yet.

Setting up Connectivity

For each MongoDB pod we created a k8s service of type LoadBalancer in order to open it for connections from the other cluster. This step needs to be done carefully in order to make sure we don’t open them to the internet and create a security risk. For that we take note of the IP address(es) of each cluster and use the loadBalancerSourceRanges directive to allow access only from the other cluster.

Note: The outgoing IP address depends on the configuration of the cloud provider. In my experience, some use the external IP of the node itself and others choose the first LoadBalancer service’s IP address. You can check it by kubectl exec ‘ing into a pod and running curl https://api.myip.com .

The service resource should look like:

apiVersion: v1

kind: Service

metadata:

name: mongo-old-0-public-ip

labels:

name: mongo-old-0-public-ip

spec:

type: LoadBalancer

loadBalancerSourceRanges:

- 1.2.3.4/32

- 1.2.3.5/32

- 1.2.3.6/32

ports:

- name: mongo

port: 27017

selector:

name: mongo

statefulset.kubernetes.io/pod-name: mongo-0

Now that we have a public IP for each pod, we need to create a proxy from each cluster to the three pods that are not in that cluster. I used a simple nginx server with the following config:

server {

listen 27017;

proxy_pass 1.2.3.4:27017;

}

The names of the pods are used by the real MongoDB pods, so we’ll make up some other name and override the hostname and subdomain properties in the proxy’s yaml file:

apiVersion: v1

kind: Pod

metadata:

name: mongo-old-0

labels:

name: mongo-old-0

role: mongo-proxy

spec:

terminationGracePeriodSeconds: 10

hostname: mongo-0

subdomain: mongo-new

containers:

- name: mongo

image: mongo_proxy:migration_old_0

ports:

- name: mongo

containerPort: 27017

The service that connects the proxy pods:

apiVersion: v1

kind: Service

metadata:

name: mongo-old

labels:

name: mongo-old

spec:

clusterIP: None

ports:

- name: mongo

port: 27017

selector:

role: mongo-proxy

We should end with 3 services and 3 proxy pods in each cluster:

In the old cluster:

Service mongo-old-0-public-ip

Service mongo-old-1-public-ip

Service mongo-old-2-public-ip

Pod with hostname mongo-0.mongo-new → forward to mongo-new-0-public-ip

Pod with hostname mongo-1.mongo-new → forward to mongo-new-1-public-ip

Pod with hostname mongo-2.mongo-new → forward to mongo-new-2-public-ip

In the new cluster:

Service mongo-new-0-public-ip

Service mongo-new-1-public-ip

Service mongo-new-2-public-ip

Pod with hostname mongo-0.mongo-old → forward to mongo-old-0-public-ip

Pod with hostname mongo-1.mongo-old → forward to mongo-old-1-public-ip

Pod with hostname mongo-2.mongo-old → forward to mongo-old-2-public-ip

Once this is done, each (real) MongoDB pod should be able to connect to every other MongoDB pod using the DNS names mongo-X.mongo-old and mongo-X.mongo-new or using the full DNS name, mongo-X.mongo-old.my-ns.svc.cluster.local , assuming the namespaces have the same name in both clusters.

Syncing Everything

Now, we can connect all the MongoDB pods to a single replica set and have them synchronize with each other. Let’s open a mongo shell to the master and run the command: rs.add("mongo-0.mongo-new") . We waited for the first pod to synchronize before adding the second in order to make sure it goes smoothly and avoid adding too much load all at once. Only after mongo-0.mongo-new finished synching and transitioned to SECONDARY state, we’ve added the second pod to the replica set and let it complete. Same for the third.

To be extra safe, we can configure the new members to be excluded from replica set leader election during the synchronization stage. This means that if the primary goes down they won’t be voting on a new primary and can’t be elected. To do this run the following commands in a mongo shell right after adding each new member, make sure to replace INDEX with the id of the new member:

cfg = rs.conf();

cfg.members[INDEX].votes = 0;

cfg.members[INDEX].priority = 0;

rs.reconfig(cfg);

Shifting the Scales

At this point we have a functioning cross-cluster replica set with 3 members in the old cluster and 3 in the new cluster. We want to move the primary to the new cluster and take down one of pods in the old cluster in order to assure the new cluster has a majority. Let’s configure the replica set again to choose a primary from the new cluster. If you changed the votes like in the example above you need to revert it.

cfg = rs.conf();

// Lower the priority of the old members

cfg.members[0].priority = 0.5;

cfg.members[1].priority = 0.5;

cfg.members[2].priority = 0.5;

// Restore the voting rights and priority of the new members

cfg.members[3].votes = 1;

cfg.members[3].priority = 1;

cfg.members[4].votes = 1;

cfg.members[4].priority = 1;

cfg.members[5].votes = 1;

cfg.members[5].priority = 1;

// Save the changes

rs.reconfig(cfg);

// Force the primary to step down and select a new one

rs.stepDown()

Next step, we scale down the old cluster’s stateful set to 2 pods. Note we only shut down the pod and not remove it from the replica set, this is important if we want to rename the pods and keep the old naming convention after we finish with the migration.

Declaring Independence

At this point we are mostly done! MongoDB is running on the new cluster, we can move the rest of the system to the new cluster and test everything is running okay. In case we have a green light we can move to the final step of disconnecting the clusters. There is only one last decision to make, do we want to keep the old hostname mongo-X.mongo-old or use the new mongo-X.mongo-new ?

If you are content with the new hostname and have configured your system to access it, then just delete the proxy pods and remove the old members from the replica set:

rs.remove("mongo-0.mongo-old")

rs.remove("mongo-1.mongo-old")

rs.remove("mongo-2.mongo-old")

Note that by deleting the proxy pods before running the commands the old cluster’s MongoDB instances are not aware of the change and can still function as a working replica set in case we need to move back. Don’t forget to remove the proxy pods there and remove the new hostnames from that replica set.

In case you want to use the old hostnames in the new cluster, first remove the proxy pods and their service. After that you need to delete the headless service of the stateful set and create a new one with the old name. You also need to restart the MongoDB pods in order for them to connect with the new hostnames. They will load the replica set configuration and figure out they are the members with the old name and still have the latest data. When all of them are up and running, they’ll successfully connect to each other and stay in sync. We can now remove the “new” members which no longer exist:

rs.remove("mongo-0.mongo-new")

rs.remove("mongo-1.mongo-new")

rs.remove("mongo-2.mongo-new")

The replica set is now disconnected between the clusters. Go over the k8s services and make sure you deleted any leftover resources we used for the migration.

One caveat: I know the title says zero downtime, and technically the migration itself was with zero downtime. This restart does introduce a downtime, but only in case you can’t change the hostnames.

Congratulations

You have migrated the MongoDB database between two Kubernetes clusters without major interruptions to the operation of the system. This process was a little slow and scary but if you followed it properly the users didn’t notice any disruptions and the feeling of satisfaction at the end is overwhelming.

I know this is a complex task and I hope this article will help you go through a migration with ease. If you face a similar challenge and would like to consult me or have any questions about it, feel free to leave a comment.

Follow us on Twitter 🐦 and Facebook 👥 and join our Facebook Group 💬.

To join our community Slack 🗣️ and read our weekly Faun topics 🗞️, click here⬇