Akka Cluster is a very powerful tool allowing to model high performance stateful applications. Quite recently we have encountered a few clients with systems combining multiple different services into a single Akka Cluster. Let’s discuss today why this is a bad idea, and why Akka is not designed for such a scenario.

Let’s imagine that you have a system composed of 20 microservices. How would it work on a single Akka Cluster?

1. Who is the seed?

First of all, Akka Cluster requires seed nodes. Those are the initial contact points which other nodes will be joining. When you are deploying a microservice number 18, then which one is it going to join? We have seen scenarios where dedicated services were created, with a sole purpose of being seeds for other services. In such a scenario your services are tightly coupled, and nothing can be in practice deployed, when the special Seed Node service is down.

If you are using Kubernetes, Mesos or other similar systems, then Akka Management extensions allow to manage seeds discovery automatically for you. To read more about that take a look at Grzegorz Kocur’s blog post Running akka-cluster on Kubernetes.

2. Cluster state — deployments, split brain resolver

Each of the Akka Cluster node has a specific current state assigned.

When the node is being intentionally shut down, then you make sure it notifies the cluster about that fact. Akka Managment with Kubernetes usually does that automatically, but when you don’t use that package, the most common scenario is that you have some custom pre-stop hook. However, if you don’t notify the cluster that this shut down is intentional, then the node will go into the unreachable state. So what happens in case in which you shut down half of your microservices in the cluster? Well, it may consider this as a split brain. It may conclude it is a network partition and depending on the algorithm of a Split Brain Resolver the result may be that the remaining working part will get shut down, assuming that second part of the cluster is working and containing the majority.

Be aware that all of the services in the cluster are tied by the cluster state, so you need to make sure that lifecycle related operations are properly executed. This also means that often you just can’t shut down any part of your system, which means that services are again, tightly coupled.

Akka Cluster has a setting named Auto-Downing, making the unreachable nodes go automatically directly to status down. However it is not recommended to use it in the production environment especially when you are using Cluster Singleton, Sharding or Akka Persistence.

3. Binary compatibility

Let’s say that you reach the moment when you would like to use a new Scala/JDK version in a new fresh microservice. However, with single Akka Cluster it is not so simple. All nodes must be binary compatible! This concerns not only Akka versions in all of the services, but also Scala and Java versions if you are still using java serialization (you shouldn't).

Having the requirement of compatible Akka versions often leads to the creation of the common project. Many people consider sharing code between services as an anti-pattern. You may say that it is only a small lib solving this one particular problem, but often it goes out of the control growing into the biggest project in the organisation.

4. At most once

Akka by default uses at-most-once message delivery semantics. It may not be the best choice for inter-service communication (depending on the project of course).

There is a special mix-in called AtLeastOnceDelivery, however you need Akka Persistence in order to be able to use it.

5. Testability

Currently, most developers use the standard untyped Actors. When you connect different services into a cluster, then usually you fetch the ActorRef of the service located in a different applications by its path. This means that the “interface” for communication does not include any type information of expected messages. You need to switch back-and-forth between repositories to see what commands can be handled. This may lead to implementation errors, but makes the test writing process much more difficult. Some people may strive to share the code fetching ActorRef's and wrap it in some traits or to introduce test Actor mocks. This, however, causes the infamous common module to grow and introduces the coupling to the system.

6. Akka team says so

Since Akka 2.5.14 documentation includes a new section “When and where to use Akka Cluster”.

In general we recommend against using Akka Cluster and actor messaging between different services because that would result in a too tight code coupling between the services and difficulties deploying these independent of each other, which is one of the main reasons for using a microservices architecture. Source: https://doc.akka.io/docs/akka/current/cluster-usage.html#when-and-where-to-use-akka-cluster

It clearly states that using Akka Cluster for inter-service communication is the wrong way and may lead to a Distributed Monolith.

Summary

I won’t deny that there are systems where different services are connected into a single Akka Cluster, they are working and making money. However, the choice of Akka Cluster as the main method of communication is not the best thing. It often leads to a tighter coupling of services, resulting often in the overhead related to development and management of deployments.

What can you do instead? Well, there are a lot of options. The good “old” REST HTTP APIs or the binary gRPC (there is even the Akka gRPC). However, you can also base your microservices architecture on messages, and use e.g. Apache Kafka.

One may ask, what about Cluster Client (feature allowing to connect to a cluster from another actor system)? Well, in Akka 2.6 it will be deprecated (superseded by Akka gRPC) and in 2.7 removed.