Horizontal Scaling (3 -> 5)

We will begin by scaling a 3-node Elasticsearch cluster to a 5-node cluster. If you don’t already have a cluster up-and-running, I recommend checking out our previous post on deploying an Elasticsearch cluster from scratch in 10 steps.

In the Nodes tab, click on the arrow corresponding to the Elasticsearch node cluster (we named it elasticsearch-production in the previous post) to open node cluster details.

Click on the node cluster details in the top right region

In the screen that appears, drag the number of nodes slider from 3 to 5 and then click on apply. Wait for the nodes to deploy.

Increase the size of the node cluster from 3 to 5

We now have five nodes available to deploy Elasticsearch service. Let’s make the necessary changes in the stackfile and redeploy.

We will change the number of minimum_master_nodes in the stack file from 2 to 3 (which is recommended for a 5-node cluster). You can read more about the optimum value of minimum_master_nodes here.

Next, we will add elasticsearch-4 and elasticsearch-5 to the list of unicast hosts so that the new nodes can be discovered using the Docker Cloud service discovery.

Save the stackfile after editing

Redeploy the stack

Now save and redeploy the stack. It can take a few minutes for the shard migration to complete and the cluster to stabilize. The status of the cluster can be checked by using the Cluster Health API (as mentioned in the Overview section of this post).

Voila! We scaled Elasticsearch to a 5-node cluster in less than 10 minutes. If you wish to scale it back, you need to start in reverse by changing the stackfile (minimum_master_nodes and unicast hosts) and then purging the unused nodes in the cluster.

Vertical Scaling (m4.large -> m4.xlarge)

Scaling the cluster vertically is slightly more involved. In this post, we will migrate the cluster from 3 m4.large instances to 3 m4.xlarge instances.

The cluster is initially in a state where all the Elasticsearch instances are running on m4.large instances.

We will first create a node cluster of size 1 composed of the type of instances you want to migrate the Elasticsearch cluster to (m4.xlarge in this case), and with the same tags as the elasticsearch service (elasticsearch, dockercloud and appbase).

Deploy a new node cluster with m4.xlarge instance type (of size 1)

We will name this cluster elasticsearch-production-vertical.

Next, we will decrease the size of the elasticsearch-production (the one with m4.large instances) from 3 to 2 and redeploy the stack.

Decrease the size of the old node cluster from 3 to 2

Redeploy the stack (2 m4.large nodes, and 1 m4.xlarge node)

Wait for the cluster to stabilize.

The cluster is now in this adjacent state.

Two of the Elasticsearch instances are running on m4.large nodes and one of them is running on a m4.xlarge node.

Next, increase the size of the elasticsearch-production-vertical cluster from 1 to 2 and decrease the size of the old one from 2 to 1 and redeploy the stack.

Increase the size of m4.xlarge node cluster from 1 to 2

Decrease the size of m4.large node cluster from 2 to 1

Redeploy the elasticsearch-stack service (1 m4.large node, and 2 m4.xlarge nodes)

Wait for the cluster to stabilize.

The cluster is now in a state with one instance of size m4.large and two instances of sizes m4.xlarge.

As a final step, we will increase the size of the elasticsearch-production-vertical cluster from 2 to 3, decrease the size of elasticsearch-production cluster from 1 to 0 and redeploy the service. The elasticsearch-production cluster can now be deleted.

Wait for the cluster to stabilize.

This is the final state of the cluster. We have now upgraded all the nodes of the Elasticsearch cluster from m4.large to m4.xlarge.

Contrasting: Horizontal vs Vertical Scaling

When scaling the nodes horizontally, we increased the cluster nodes in a single operation and then changed the stackfile to reflect the updated cluster nodes.

In contrast, when scaling vertically, we create a new node cluster of the intended size (m4.xlarge) and follow this rinse-and-repeat pattern of:

Adding a node to the new (m4.xlarge instance) node cluster

Deleting a node from the old (m4.large instance) node cluster

Redeploying the stack and wait for the cluster to stabilize

Rinse-and-repeat pattern of node cluster configuration changes.

Since we don’t change the number of overall nodes where Elasticsearch is being deployed, the stackfile never changes.

Cross Cloud Migration

Migrating a service from one cloud to another is a big headache, often involving service downtime and consuming days of efforts.

Being agnostic to the underlying hardware and network, a containerized deployment shines here. Migrating from one cloud to another is exactly similar to vertical scaling, yes — you heard it right!

We will now migrate the elasticsearch-production-vertical cluster from AWS EC2 to Digital Ocean. First things first, link your Digital Ocean account to Docker Cloud.

Link Digital Ocean account

Create a new node cluster of size 1 with Digital Ocean.

Create a node cluster with Digital Ocean as the provider

Decrease the size of the existing cluster (elasticsearch-production-vertical) by one and redeploy.

Wait for cluster to stabilize.

Now increase the size of DO cluster from 1 to 2, decrease the size of AWS cluster from 2 to 1 and redeploy.

Wait for cluster to stabilize.

Finally, increase the size of DO cluster from 2 to 3, decrease the size of AWS cluster from 1 to 0 and redeploy the stack. You can now delete the AWS node cluster.

Wait for cluster to stabilize.

Voila! We live migrated the Elasticsearch cluster from AWS to Digital Ocean.

The rinse-and-repeat timeline showing cross migration (it’s exactly like the one for vertical scaling)

Contrasting our migration approach v/s Flocker

In our approach, we use Elasticsearch’s native capability to distribute as a way to scale, replace and migrate individual nodes. On the other hand, Flocker from ClusterHQ is a data volume manager for containers. Tools like Flocker have a notion of data volume, which is portable and moves along with the container. We think this approach is great for general purpose apps. But distributed systems (like Elasticsearch) already have a built-in replication and distribution mechanism, and work better by themselves.

Conclusion

In this post, we talked about how to:

scale a 3-node Elasticsearch cluster to a 5-node cluster, change the node types (from m4.large nodes to m4.xlarge nodes), live migrate entire Elasticsearch cluster from AWS to Digital Ocean with minimal downtime.

And contrasted the differences between horizontal / vertical scaling, and how our approach compares with flocker’s. This concludes our introduction to deploying and fully operating a basic elasticsearch cluster in production.

If you liked this post, you will love our upcoming posts on deploying the Elasticsearch-Logstash-Kibana(ELK) stack with Docker Cloud, covering more complex Elasticsearch topologies and with other orchestration tools (like Kubernetes and Rancher).