RabbitMQ 4.x & Classic Queue Conversion

[Kris Reese]

Would the release of RabbitMQ 4.x include an option during the upgrade process to convert existing Classic Queues over to Quorum Queues?

[Luke Bakken]

Hello!

We're working on options! It's not quite as simple as flipping a switch.

Thanks,

Luke

[Luke Bakken]

To expand on that a bit... ideally users will migrate to quorum queues now because they are superior in every way when compared to classic mirrored queues.

Since quorum queues aren't created via policy like classic mirrored queues there's a bit of a mismatch there when it comes to a "magic button" upgrade.

There are several more subtle issues that some of my teammates might want to comment on here as well.

Advice: upgrade RabbitMQ and Erlang, and start migrating to quorum queues now!

Thanks,

Luke

[Michael Klishin]

The most complicated parts of such an upgrade would be

* Queues with massive backlogs that would take a long time to migrate

* Concurrent actions by clients

* The fact that two queues with the same name cannot exist in a virtual host

so any strategy of "automagic" migration will fail flat on its face in some cases, or violate someone's assumptions.

Instead of waiting for a magic button I'd start migrating specific queues or groups of queues to QQs today. There is a migration

guide in the works, time will tell if it would be made public or internal to paying customers.

[Rin Kuryloski]

One of the basic challenges is that queue types are chosen by and transparent to the client.

From https://www.rabbitmq.com/tutorials/amqp-concepts.html: "AMQP 0-9-1 is a programmable protocol in the sense that AMQP 0-9-1 entities and routing schemes are primarily defined by applications themselves, not a broker administrator. Accordingly, provision is made for protocol operations that declare queues and exchanges, define bindings between them, subscribe to queues and so on."

All the queue types, aside from streams, do have lots of commonalities. But, it's not as if there is a single queue abstraction and whether it's classic or quorum underneath is an implementation detail of the broker. Therefore changing queue types can't be guaranteed to be a non-breaking change for all clients. We have worked to bring feature parity, to the extent possible, to quorum queues from mirrored classic queues so that is possible for the maximal number of user to upgrade. But, there are also practical considerations, such as what should happen to queue availability during a queue type migration, if the magic button existed? An empty queue would certainly be the easiest (and quickest) but a full queue would need to either be "paused" during the migration, or might need something like a temporary internal federation. A should a network partition occur during migration, things get even more complicated.

Given that a number of the possible tradeoffs worth making during an upgrade might be specific to an application, it's a good idea to give some though to your own upgrade plan. The more that users can share, the greater likelihood that common cases will become evident, and may increase the chances that any tools possibly build to ease or automate migration will apply to the greatest number of RabbtMQ users.

[Kris Reese]

Thank you, all!  We've had some developers migrate to QQ's. The process has been that they refactor their code to declare a QQ, and before deploying the refactored version, we coordinate with them to stop the application, delete the classic queues, deploy the app and let the refactored code declare the QQ.  Seems to be working for us in that either the queues are already zero, or the devs are okay with trashing any remaining messages.

On the other hand, we have some who are waiting to see if there will be a magic button.  In this case, they'd still need to refactor their code from declaring a classic queue, right?

[Luke Bakken]

Responses in-line...

On Friday, December 9, 2022 at 7:01:16 AM UTC-8 ktr...@gmail.com wrote:

Thank you, all!  We've had some developers migrate to QQ's. The process has been that they refactor their code to declare a QQ, and before deploying the refactored version, we coordinate with them to stop the application, delete the classic queues, deploy the app and let the refactored code declare the QQ.  Seems to be working for us in that either the queues are already zero, or the devs are okay with trashing any remaining messages.

That's a great plan.

 

On the other hand, we have some who are waiting to see if there will be a magic button.  In this case, they'd still need to refactor their code from declaring a classic queue, right?

"It depends". Do they expect the queues to be mirrored or not? RabbitMQ 4.0 will still have non-mirrored "classic" queues as those have plenty of valid use-cases, and we've made impressive improvements to them in recent releases. 

[Kris Reese]

Thank you for all of the info!  This will help us plan as we're rather up-to-date on our version, and we're working to get ahead of a 4.x release and asking devs to convert over to QQ's.

We are mirroring queues via policy ha-all to avoid message loss.  I assume this policy applies no matter how devs declare a queue in code?  Meaning,  a developer can't somehow specify the creation of a non-mirror queue and avoid the policy from applying itself to that declared queue?  If that's the case, then clearly for us, non-mirrored classic queues aren't in use.  I'd be interested in understanding some of the use cases where a non-mirrored queue would be the best choice though!