In this post I am going to describe some specifics of RabbitMQ broker on how it deals with broken messages and tell about some solutions.

All examples are based on Enqueue PHP Messaging library

By default RabbitMQ server does not wait for a consumer’s acknowledgment . The message is removed from the queue as soon as it is delivered to the consumer. Hence the message can be lost if the consumer dies while processing the message. It may be okay for some cases but in general it is better to turn on message acknowledgment mode.

Here’s the most important part about the mode:

If a consumer dies (its channel is closed, connection is closed, or TCP connection is lost) without sending an ack, RabbitMQ will understand that a message wasn’t processed fully and will re-queue it.

and

There aren’t any message timeouts; RabbitMQ will redeliver the message when the consumer dies. It’s fine even if processing a message takes a very, very long time.

Let’s imagine that our consumer has a bug and it is not able to process a message. It throws an exception all the time it gets this particular message. The unacknowledged message is returned to the head of the queue, not to the tail as you might think. Here are the consequences of this assertion:

If you have a single consumer it will receive the same message again and again and will never be able to process messages that are piled up after this buggy one. Hence because of one buggy message you may have thousands of pending messages in the queue.

The message is redelivered to the consumer with immensely high speed. The broker sends it again and again. It reflects on CPU consumption and the server load in general.

The process dies while processing a failed message and it has to be restarted. It takes resources and time too.

The logs are filled up very quickly.

There is a way to find out whether it is a new message or a re-delivered one. Here’s the most important part from the official doc:

If a message is delivered to a consumer and then requeued (because it was not acknowledged before the consumer connection dropped, for example) then RabbitMQ will set the redeliveredflag on it when it is delivered again

Now let’s see how we can make use of this redelivery flag in our PHP application.

Consider this example:

Having that in mind we can requeue a redelivered message. It will be added to the tail of the queue. It definitely solves the first problem with piled up messages but does not solve the second one.

To solve the second issue we need a kind of delayed redelivery but unfortunately RabbitMQ does not support that out of the box.

One “solution” (I’d consider it a workaround) to this problem is to use a mix of message TTL and Dead Letter Exchange as proposed by Kien Nguyen here. The mechanics is well described there so there is no need to repeat. I’ll skip that part and show you how it can be implemented in PHP:

Another solution is to install a RabbitMQ rabbitmq_delayed_message_exchange plugin. The plugin adds a delay exchange, whenever a message is sent to that kind of exchange with x-delay header It will be delayed for that period of time and only after that time passes it is routed to a queue.

Please note: that the delayed messages are not visible. You are not able to find out the number of delayed message nor delete them. They are invisible and when the time comes they show up out of nowhere.

Here is a PHP usage example:

QoS. pre_fetch_count. Update from 10–11–2018

The pre_fetch_count bigger than one can cause message redelivery. For example, you have pre_fetch_count equals two. It means a broker schedule two messages from a queue to be processed by a consumer. The messages are sent to the consumer at once. It starts processing the first one and fails (the socket is unexpectedly closed). In this case, two messages are redelivered event through the second has not been ever processed.

Conclusion:

There are some problems with the redelivery and we highlighted them. Than we went through several solutions that address the issues: from the simplest one (redelivery + requeue) to advanced ones (delay plugin). The delay plugin approach is the clearest and straightforward solution and I’d personally recommend to use it. I’d suggest to use the Enqueue library since it already supports the delay plugin and you won’t have to deal with most of the stuff manually.

You can also visit our corporate blog to learn more about up-to-date IT trends and download some useful insights.