How to use Apache Kafka on Azure in a Microservices Architecture. Image from: www.arctiq.ca

If you have followed me recently I have been writing a series of articles around Microservices Architecture and Implementation. I hope you read the First , Second, Third, Forth and Fifth part of this Microservices Implementation Journey and found them useful and if you didn’t I recommend you do!

Here are the links for the previous parts:

Part 1: https://koukia.ca/a-microservices-implementation-journey-part-1-9f6471fe917

Part 2: https://koukia.ca/a-microservices-implementation-journey-part-2-10c422a4d402

Part 3: https://koukia.ca/a-microservices-implementation-journey-part-3-50f030ba6bb5

Part 4: https://koukia.ca/a-microservices-implementation-journey-part-4-9c19a16385e9

Part 5: https://koukia.ca/a-microservices-implementation-journey-part-5-d7d1b9d441e7

If you followed along, one of the main components of a Microservices Architecture, is Service Bus or Event Store components that enables our decoupled services to interact with each other in an asynchronous way.

There are several alternative technologies out there that we can use for this purpose, that I mentioned them in the first part of this Microservices series.

The focus of this part of the series will be to show case Apache Kafka and use it as our Pub/Sub communication channel in our Microservices echo-system.

And to make things even more interesting I will use Azure to host my Apache Kafka instance and will use an Asp.Net Web API to publish and subscribe to events from this Kafka cluster hosted in Azure.

If you are interested, get your cup of coffee or whatever your poison is and read on!

What is Apache Kafka?

Before we get started with installation and other fun stuff, let’s see what Kafka is.

Apache Kafka® is a distributed streaming platform.

Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. Image from: https://kafka.apache.org/

Tell me more about Apache Kafka

Kafka has 3 main features:

A publisher and Subscriber mechanism for streams of data, similar to an Event Store or an Enterprise Service Bus (Though Kafka’s approach for doing Event Sourcing it is somewhat different than Service Bus or some Event Stores)

Kafka stores the streams of events/data in a very fault tolerant and durable way.

Kafka processes streams of data as they occur in the Kafka cluster.

Can I use Kafka for any kind of application?

I think it can be used for any system that needs some sort of Messaging or Event Streaming but it is typically used in two types of architectures:

If you need to move streams of data around in your system in real-time between applications Kafka is your technoly.

If you need to identify events that happens in your system and then do something or trigger some workflow based on those events or streams of data, Kafka is your technology.

How to use Apache Kafka on Azure in a Microservices Architecture

Without further due, let’s jump into Azure and stand up our Kafka Cluster.

In Azure Kafka cluster is deployed as part of the Azure HDInsight service.

Side note to see what else Azure HDInsight can do:

Azure HDInsight is a fully-managed cloud service that makes it easy, fast, and cost-effective to process massive amounts of data. Use popular open-source frameworks such as Hadoop, Spark, Hive, LLAP, Kafka, Storm, R & more. Azure HDInsight enables a broad range of scenarios such as ETL, Data Warehousing, Machine Learning, IoT and more.

So to get started, I sign in to Azure portal, and from the “create a Resource” menu search for “HDInsight”, and select the first result from the Marketplace and click “Create”: