Scaling consumers with consistent hash exchange

[Tom Leavy]

Hello all,

I'm trying to solve for a problem we have with the consistent hash exchange in combination with the specific application scaling logic we require.

1. All messages within a specific hash must be processed in order by a consumer.
2. The process that consumes queues must be able to scale up by spawning new instances when load exceeds a certain threshold.

3. The process that consumes queues must be able to be scaled down when traffic is lower without breaking rule #1

Right now design we have is that each time a consumer comes online it binds a durable queue to the exchange that matches its instance id (number between 1-N). This works fine as long as the instance count is always scaling up. Messages are properly rebalanced across queues and each hash is bound to a specific consumer to keep all message processing in order.

The problem is that when we want to scale down due to low traffic, we need a way to make sure that messages in the queues that are now being dropped are properly routed back to the exchange and then to whatever queue now owns the hashes those messages belong to post rebalancing. It is also important that those existing messages be consumed in order before any new messages that arrive on that same hash are processed in order to maintain our first requirement.

Any ideas on how to solve for this? Is this something we can solve for out of the box, or with a custom plugin?