Say Hi to Delivery Boy!

A simple, convention-based library for publishing messages to Kafka from Rails.

Hot off the heels of releasing Racecar, our easy-to-use framework for running Kafka consumers in Ruby, we’re ready to release yet another Kafka-related library!

This time, it’s all about message delivery.

Up until now, if you wanted to produce messages to Kafka from a Rails application, you’d need to write some boilerplate code: instantiating the ruby-kafka client with your desired configuration, setting up a producer instance with the right message delivery policy, and making sure to shut things down properly upon exit, including making sure to deliver any pending messages.

It was never a lot of work, but it was still an impediment to getting started, and there were too many chances to mess things up. That’s why we’re releasing Delivery Boy, a simple library that helps you publish messages to Kafka. It reads configuration from a YAML file (or environment variables, if you’re into that) and handles setting up thread-safe Kafka producer instances for you.

Getting started

Add this to your Gemfile:

gem "delivery_boy"

Then run bundle install followed by bundle exec rails generate delivery_boy:install . This will generate a config file at config/delivery_boy.yml. If your Kafka broker is running locally, this should work out of the box. Otherwise change the file to point at one or more of your brokers.

Once this is done, you can start publishing messages to Kafka immediately! Here’s an example:

This will push the pending message to a background thread that handles message deliveries automatically. Doing this is important if you want to insulate your application against Kafka broker failures: while it is possible to deliver messages to Kafka directly using the deliver method, doing so would mean that a delivery error would blow up your whole application. We recommend using deliver_async unless you’re absolutely sure that it’s okay for your Rails code to get an exception — or maybe hang for a while — when Kafka is having issues. Better safe than sorry.

At Zendesk, we’re already using DeliveryBoy at scale, producing between 1,000 and 2,000 messages every second in our global deployment.

If you want to read more about DeliveryBoy, head over to its GitHub page!