Changing Queue Parameters in a Production System

[Devin Riley]

Hi,

I'm using RabbitMQ as a message queue in a service-oriented architecture, where many separate web services publish messages to RabbitMQ. Those queues are in turn subscribed to by various consumers, which perform background work.

I'd like to change various backwards-incompatible parameters of the queues that are being published to by the web services (I'm adding a dead-letter exchange, a dead-letter-routing-key to queues, and setting a per-queue message TTL on some of the queues). Whats the best way to change these queue parameters with minimal downtime and zero message loss, and without coordinating the change across the web service publishers? Just to give some details on this system, we use RabbitMQ for small-load but highly-important messages. We're running just a single node, and we've acked about 20K jobs from our work queues over the last few weeks.

The distributed web services publish to queues passively, so they don't declare queues themselves which should mean I can change the parameters (aside from routing-key) without them having any problems.

I asked this question on StackOverflow, so you can see a couple possible solutions we discussed there (http://stackoverflow.com/questions/25274182/rabbitmq-change-queue-parameters-on-a-production-system), but please add your own thoughts on the best practices around this.

Thanks in advance,
Devin

[Michael Klishin]

On 14 August 2014 at 04:52:47, Devin Riley (devinr...@gmail.com) wrote:
> > Whats the best way to change these queue parameters with minimal
> downtime and zero message loss, and without coordinating the
> change across the web service publishers?

It depends on what parameter that is. If it is durability, the auto-delete
flag or exclusiveness, the queue has to be deleted and re-declared.

Additional arguments (the arguments map) can be controlled via policies.

Publishers rarely need to declare queues, see
https://groups.google.com/d/msg/rabbitmq-users/HhCZgE9_y6E/zwZUM2dLMvcJ
for discussion.

One migration strategy that I've used multiple times is

 * Use a version suffix in queue name
 * Make new service versions use a separate queue with the new parameters and updated version number
 * Bind them in a way that makes the same messages go to them (when it is not possible, you almost always can get away by using an intermediate exchange and exchange-to-exchange bindings)
 * Upgrade all service instances, drain messages in the old queues
 * Move permanently to the new version
 * Delete extra bindings, etc, as needed 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Bogdan Padalko]

One migration strategy that I've used multiple times is

 * Bind them in a way that makes the same messages go to them (when it is not possible, you almost always can get away by using an intermediate exchange and exchange-to-exchange bindings)

The cons here are:

• messages can get duplicated to multiple queues, which under certain circumstances may not be acceptable;
• also the dark side here, especially when multiple application uses the same RabbitMQ server are dead letter and alternate exchanges policy, which may route messages directly to old queue via default exchange.