Exchanges

AMQP defines the concept of exchanges. Exchanges can be thought of as routers. When a message is published to an exchange, the exchange determines which queues should the message be delivered to. It’s important to note that it’s impossible to put a message directly into a queue. Even if a message is published directly to a queue, a temporary exchange will be created to deliver it to the queue.

There are four types of exchanges supported by RabbitMQ.

The most commonly used exchange type is the direct exchange. It directly delivers all messages to a single queue bound to it. It’s a 1-on-1 mapping of an exchange to a queue. If applied to a chat application, a direct exchange would deliver messages from a chat room to a single user.

Animation of a direct exchange being utilized in a chat app

Another kind of exchange is the fan-out exchange. They deliver messages to all queues bound to them. It’s a 1-to-many mapping of an exchange to multiple queues. In the example of the chat application, a fan-out exchange would be used to send a message to all users.

Animation of a fan-out exchange being utilized in a chat app

Then there are topic exchanges. They deliver messages published to them to a bound queue based on the messages tag/topic and the queue’s bound topic. In the example of the chat application, a topic exchange would be used to direct messages to their corresponding user.

Animation of a topic exchange being utilized in a chat app

Finally, there are header exchanges. They are a step up from topic exchanges. Instead of looking at a topic, also called a routing key, they look at the message’s headers to determine where a message should be delivered. Messages in RabbitMQ can have additional attributes associated with them — called headers. Headers determine different behaviors when handling messages. E.g. an “x-match” header indicates to the exchange that either any or all headers have to match a value for it to get routed to a queue. There is also the “reply-to” header which indicates where the result of processing a message should be published to.

Utilizing exchanges gives many advantages — exactly once delivery, performance, and ease of deprecation. Utilizing an exchange to deliver your messages is much faster and more reliable than using Ruby to handle that logic. There is also the pragmatic reason of not having code to maintain. The logic is handled by RabbitMQ, you only have to configure it (which can be done through code). Exchange-exchange and exchange-queue bindings can be changed on-the-fly by any client which enables one application to change the behavior of other services. Personally, exchanges have helped me deploy applications with little-to-no downtime and deprecate services without having to change other services.

Special features

RabbitMQ adds it’s own magic on top of AMQP. I have already mentioned header exchanges, which are a non-standard AMQP feature. A feature I personally use a lot is “direct reply-to”. Direct reply-to is a form of synchronous communication between a producer and a consumer. It enables a producer to publish a message and wait for a consumer to process it and return the result directly to the producer. It’s useful when the result of a message is used in further processes. E.g. IoT devices usually log a heartbeat signal to their server to indicate that they are connected and configured correctly. If we have a smart lock, we can process its heartbeat asynchronously since the result isn’t really important for the server nor for the device. But a pin check is important and should be handled synchronously to avoid access permission errors caused by stale data.

Animation of utilizing a direct reply-to message

Another feature that I often utilize is “dead lettering”. It allows for a message to be re-queued automatically in case it gets rejected from an exchange or queue. This feature is useful for error handling, exponential back-off, scheduled message processing …

Animation of utilizing a dead letter queue

“Alternate exchange” is a useful feature for deprecating services. It specifies to which exchange a message should get sent to in the case that the primary exchange rejects it.

Animation of utilizing an alternate exchange

Then there are “priority queues” and “priority consumers”. Priority queue are CS standard priority queues. Meaning that messages in the queue have a priority (ranging from 0 to 255), messages with higher priority get processed first. Which is useful for the same reasons as “direct reply-to”, but it’s asynchronous. While priority consumers are a form of fail-over. Priority consumers will be served messages if they are active. In the case that all priority consumers are inactive, other consumers will get served messages.

Finally there is “TTL”. It specifies how long a message lives. If a message outlives its TTL it’s automatically rejected from the queue. Though, this feature comes with a caveat — this rule can only be enforced for the message at the front of the queue. E.g. if you put two messages in a queue the first with a TTL of 300 and the second with a TTL of 100. Both would be in the queue until the one with a TTL 300 expires, because it is in front. The moment it expires, the second message is at the front and automatically expires since it’s TTL has passed. This seems harmless, but can cause a lot of problems when combined with dead lettering to achieve e.g. offset delivery.

Animation of TTL messages combined with dead lettering

Plugins

For me, this is the most important feature of RabbitMQ. With plugins you can add any functionality you want to RabbitMQ. The best example of this is the management console which is a plugin, and must be enabled before use.

Through plugins, RabbitMQ supports not only AMQP, but STOMP, MQTT and WebSockets as communication protocols.

Then there is the Federation plugin. It enables RabbitMQ to run several isolated clusters or instances which can communicate with one-another. It is similar to the way that Mastodon works. All users, no matter which server they signed up to, can communicate with one-another. Federations are useful for handling large workloads. E.g. if you handle logs from a lot of different machines through RabbitMQ, that can be handled by one federated cluster, while everything else is handled by another federated cluster. That way you can scale those two cluster independently depending on their workload, and you avoid the noisy neighbor problem (when a highly taxed service slows the whole system down).

Example RabbitMQ Federation for an IoT application

Conclusion

Replacing Sidekiq with RabbitMQ provides many advantages when it comes to debuggability, scaling, fault tolerance and memory consumption. It supports multiple industry-standard message queue protocols and can be used as a drop in replacement for other “background worker” libraries.

If you need a queue that guarantees job execution and persistence, go with RabbitMQ instead of Sidekiq. There are some features that are missing in Rabbit, like cron jobs and unique jobs, but they can be added by the clients. RabbitMQ offers a plethora of features which, if not useful at first, will become useful later as they will help grow a monolith to a services oriented architecture.

To get started take a look at projects like Sneakers (background jobs) and Bunny (AMQP client), read through the basic concepts page, and lastly there is the manual. If you are using Ruby on Rails, Sneakers integrates with ActiveJob which eases the transition.

Sidekiq isn’t useless! If your project doesn’t require execution or persistence guarantees, or if you hold a Sidekiq Pro license, I would recommend you stick with it for the time being. While I disagree with hiding essential features (like guaranteed execution, and rolling restarts) behind a paywall, a Pro license offers support and additional business oriented features which you won’t get with a self-hosted RabbitMQ instance.