Alpakka Kafka connector — an open-source Reactive Enterprise Integration library for Java and Scala

Interview with Krzysztof Ciesielski

Alpakka Kafka is an open-source initiative to implement stream-aware and reactive integration pipelines for Java and Scala. Currently, it’s being developed by many individual contributors and supported by people from Lightbend. The idea for it emerged in one of the SoftwareMill-ers’ heads a few years ago. Originally it was created as Reactive Kafka. We asked Krzysiek Ciesielski — the first author of the library about its origins.

1. Could you briefly describe what does Alpakka Kafka library do — what problem is this library trying to solve?

It allows consuming/producing from Kafka with Akka Streams, leveraging the reactive interface of this streaming library, its backpressure, and resource safety. It hides a lot of complexity, especially when your streaming logic is non-trivial like sub-streaming per partition and handling commits in custom ways.

2. How did you or your team come up with an idea for this library?

We realized that streaming from Kafka (version 0.8 at that time) with its pure Java API was quite painful, at the same time we were starting to use Akka Streams in our projects and wanted to have hassle-free integration.

3. Why did you decide to engage in developing this open-source project?

My engagement was actually the very inception of Alpakka Kafka, called Reactive Kafka at that time. I wrote the initial version to experiment, then some feedback started to appear and it turned out to be actually useful.

4. Is there anything specific that you learned through working on this project?

I learned a lot about collaboration in a truly open-source project. Doing code reviews, handling incoming issues and dealing with documentation requires good communication. I also acquired extra sensitivity for the usability and backward compatibility of public APIs. Moreover, I had to write some performance tests, which was a pretty fresh topic for me at that time. Finally, when the Lightbend team took this project under its wings, I had an opportunity to learn about some interesting details of Akka internals and optimizations, not to mention their exemplary communication skills in a truly distributed team.

5. Do you know what are the future plans for the project’s development?

There are many upgrades on their way. The most important one is switching to a new Kafka client, adding support for partitioned transactional source, and adding user-facing API for synchronous partition assignment handler.

6. What would your advice for people willing to start their adventure with OSS be?

Tame your inner perfectionist. It’s better to publish a somewhat satisfying pull request, than to improve it forever fearing criticism in code review. Just put it out there and update it according to comments! You are not “bothering” anybody — any kind of update matters. You can start with simple fixes and updates in tests or documentation. Even small typo fix is much welcome, with such simple contributions you’ll build up your confidence and motivation.