Series content

This series creates the same Scalable Microservice Application using different technologies:

(this article) Scalable Serverless Microservice Demo using AWS Lambda Kinesis Scalable Serverless Microservice Demo using Knative and Kafka (planned)

What’s this about?

This describes a Highly Scalable Microservice Demo application using Kubernetes, Istio and Kafka. Through synchronous REST API calls it’s possible to create users. Inside, all communication is done asynchronously through Kafka.

Image 1: Architecture overview

The Kafka consumer/producer UserApprovalService is automatically scaled (HPA) based on how many unhandled messages are in the Kafka topic. There is also a Node/Cluster scaler in place.

We will scale up to 23000 Kafka events per second, 11 Kubernetes nodes and 280 pods.

Image 2: Results overview

The application is completely scripted with the help of Terraform and can be run using one single command.

Technology stack:

Terraform

(Azure) K8s, MongoDB, Container Registry

(ConfluentCloud) Kafka

Istio

Grafana, Prometheus, Kafka Exporter, Kiali

Python, Go

Repository

https://github.com/wuestkamp/scalable-microservice-demo

Check the readme for instructions on how to run it yourself.

Architecture

Image 3: Architecture

We have three microservices:

OperationService (Python): receives synchronous REST requests and converts these into asynchronous events. Keeps track of requests as “operations” and stores these in its own database.

(Python): receives synchronous REST requests and converts these into asynchronous events. Keeps track of requests as “operations” and stores these in its own database. UserService (Python): handles user creation and stores these in its own database.

(Python): handles user creation and stores these in its own database. UserApprovalServer (Go): can approve/deny a user, stateless.

The Kafka cluster is managed by ConfluentCloud, the Mongo databases and the K8s cluster are managed by Azure.

Database per Service Pattern

We don’t use one large database which multiple services share, every service has its own database if it's stateful. We still only have one MongoDB database server, but on that one server can exist multiple databases. Microservices can share the same database server if they use the same type/version. Read more here.

Asynchronous communication

The three microservices communicate asynchronously with each other, there is no direct synchronous connection. One advantage of async is loose coupling. If the UserApprovalService is down for some time, the request doesn’t fail but just takes longer till the user gets approved. Hence when using async communication there is no need for implementing retries or circuit breakers.

Message Workflow

Image 4: Message workflow

Image 4 shows the messages produced and consumed. The UserService consumes user-create messages, creates “pending-approval” users stored in MongoDB and produces the message user-approve .

Once it receives the user-approve-response message from UserApprovalService, it updates the user to “approved” or “not-approved” and produces the user-create-response message, which will be received by the OperationService which will update the operation status to “completed”.

SAGA Pattern

When you work with one large (MySQL) relational database you can simply wrap your actions in database transactions. The SAGA pattern can be used to implement ACID like transactions for actions across multiple microservices.

In Image 4 the UserService could be considered as the orchestrator of the SAGA user-create. Because it coordinates the user creation through producing and consuming various messages. In this example, only one more service (UserApprovalService) is involved, but this could get more complex with many more services.

A SAGA can be compared to and implemented as a State Machine. To read more about the SAGA pattern and the difference between Orchestration and Choreography here: https://microservices.io/patterns/data/saga.html

Synchronous <-> Asynchronous conversion

Image 5: Sync to Async conversion

(1) Image 5 shows that at first a synchronous REST call is made to the OperationService to create a new operation, in this case “user-create”. The OperationService emits an asynchronous message and then immediately returns the new operation in a pending state.

(2) The returned operation contains a UUID which can then be used to periodically fetch the current state of that operation. The operation will be updated based on further asynchronous requests made by other services.

Scaling based on Kafka message count

Kubernetes Cluster Scaling is configured with Terraform on Azure. We also have an HPA (Horizontal Pod Autoscaler) on the UserApprovalService deployment.

The HPA listens to a custom metric which provides information on how many messages are not yet processed in the Kafka topic user-approve. If there are messages queued up we spin up more pods.

The UserApprovalService sleeps 200 milliseconds after processing a message. This means it will lag behind if it’s the only instance and new messages coming in constantly.

Monitoring & Metrics

We use Prometheus and Grafana to visualise what’s happening.

Kafka metrics

To get metrics from Kafka we use the kafka_exporter which makes these available in Prometheus and by that Grafana. We implemented the kafka_exporter as sidecar in every Pod of the UserApprovalService so that the metrics can be used from/for every single Pod.

To get these Kafka Prometheus metrics usable as K8s custom metrics (required for HPA) we use the k8s-prometheus-adapter.

# confirm install

kubectl api-versions | grep "custom.metrics" # list Kafka topic metrics

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq # list Kafka topic metrics for every pod

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pod/*/kafka_consumergroup_lag" | jq

More details in the project’s prometheus-adapter.yaml . Now we’re able to use these Kafka metrics for K8s HPA.

Istio metrics / Kiali

Kiali works fantastic with Istio and instantly provides us with an overview of what's happening:

Image 6: Kiali network

In image 6 we see the REST requests hitting Istio Gateway and then OperationService. All other communication is going through “PassthroughCluster” which is the external managed Kafka. We can also see that the kafka-exporter is communicating with Kafka to gather metrics.

As of today, Istio is not able to manage Kafka traffic in more detail, it handles it as TCP. Envoy seems to be able to do this already, which means Istio will follow. We might then also see advances in Kiali, like displaying the number of messages per second on edges.

See more here in the Twitter thread of Joel Takvorian where he managed to include the Kafka nodes into the Kiali service diagram.

In Action

Now the fun begins.

UserApprovalService lagging behind

Without the HPA enabled, we create ~60 new events per second.