Horizontally scaling a queue

[Rud Wangrungarun]

Hello. I have a question about horizontal scalability of a queue. Currently we have 1 queue with 4 consumers on different node (they’re competing, which unfortunately not what I want because that make consumption out-of-order but found out later). Now things start to grow and the size of pending messages grow. We want to parallelize queue consumption while maintaining the order of the incoming message and found that consistent-hash exchange along with single active consumer to achieve partial order might be the solution.

I then found out that it’s quite difficult with RabbitMQ since you need to somehow manage the assignment of the queue to the partition. This would call for some state store (like zookeeper) and coordinator. There seems to be implementations like Apache Helix or Rebalanser but I’m on node.js environment so they’re not an option. Also I don’t want to add more complexity and component to maintain.

One idea to emulate balanced assignment is to have each consumers bind to all queues then enable SAC. With expectation that each of the queue will eventually be consumed by at least 1 consumer. But I’m not sure this is the right use-case of SAC and curious if this will even work.

Another idea is to have simple static mapping between each queue and consumer but it need to be manually maintained and manually scale up or down which will be a pain.

I’m experienced in Kafka and new to RabbitMQ and scaling like this is natural with Kafka so my mental model might still get stuck instead of doing it in RabbitMQ way. In this situation, what would be the possible solution to achieve the scaling of the queue consumption while still maintain partial ordering?

Thanks in advance,
Rud

[Karl Nilsson]

Hi,

As you've observed there is no coordination / rebalancing layer in rabbitmq that operates over queues so the options you have explored are the ones that are available.

Binding multiple queues configured with SAC to a consistent hashing exchange is the right approach for this kind of mostly ordered processing. We do appreciate that to do this successfully you need to configure the consuming applications to choose which queue(s) to consume from. The consumers still need to know about the queues that are available anyway but you need to find an approach where the select which of the queues should be their primary queue and which they should run as backup consumers. 

SAC operates on a first come first serve basis which means that it is easy for a single consumer to end up consuming from all queues if it starts up a little before the other consumers. 

To mitigate this if you could configure a "home" queue per consumer it could consume from this queue first and then wait a bit before it consumes from the other to allow other consumers in and perhaps end up with a better distribution. It won't be perfect and I hope that in the future we could provide extended coordination facilities over queues in the broker but for now that is what we have.

The last alternative would be for you to use RabbitMQ super streams which do provide broker side consumer groups / rebalancing when you use the dedicated stream protocol.

https://blog.rabbitmq.com/posts/2022/07/rabbitmq-3-11-feature-preview-super-streams

Cheers

Karl

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/9c4d52be-8650-44c2-af6a-b2faeaf5978fn%40googlegroups.com.

--

Karl Nilsson

[Rud Wangrungarun]

Hi Karl,

Thank you for the prompt response. I'll have a look on implementation details using SAC with home queue.

Best regards,

Rud