RabbitMQ double binding -- duplicate data

[Hisham Najem]

System architecture:

I have a RabbitMQ node with unallocated and allocated exchanges and queues. I have, for the sake of example 2 consuming microservices with n instances each and a broker utility microservice which is responsible for allocating (binding consumers), listening for incoming data messages, and other management functionality such as managing consumer pods.

Upon initialization, each consumer tells the broker utility microservice that it is ready to start consuming messages. In response, the broker utility will dedicate -- allocate a directed queue with routing key X (depending on message) for the consumer.

Scenario:

Message with routing key X comes in for microservice Y and if it is able to be routed by an allocated exchange to a dedicated queue then send it off to the proper consumer and we are happy. Otherwise we have a new routing key that has to be allocated and so put it through the unallocated exchange (fanout) to all pods within microservice Y and if the pod is "not consuming messages" -- i.e. is available to be allocated then the pod is allocated and a dedicated queue is attached to the pod. Now any future messages with routing key X will just go directly to the allocated pod.

Problem:

If two messages with the same routing key come in within a small time interval (in ms) then they are both processed as "unallocated" and so both are attached to two different pods and so future messages with the same routing key are duplicated amongst the two pods. There simply isn't enough time between processing both of these messages to register the routing key as "allocated".

Working solution (want to get rid of this):

Use a hazelcast lock to lock on the routing key. This puts a heavy strain on the system.

Reference diagram -- see attached

Is there a native way to lock on RabbitMQ routing keys? Is there a topology I can set up (perhaps exchange to exchange) which will fix this issue? Is there anything on the RabbitMQ side I can do? Can we set up unique routing keys (i.e. 1 queue per routing key)?

[Michal Kuratczyk]

Hi,

I'm not sure I fully understand what you are trying to accomplish but perhaps Single Active Consumer is what you need? https://www.rabbitmq.com/consumers.html#single-active-consumer

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/6f8f9c87-64e9-449a-bbf4-dc2b3329063cn%40googlegroups.com.

--

Michał

RabbitMQ team

[Hisham Najem]

Thank you for your response Michal,

Single active consumer isn't quite what I'm looking for. I'm trying to achieve a single queue per routing key. Under normal operation if message 1 comes in with routing key X then a direct queue is made and binded in an exchange with that routing key X. Subsequent messages with the same routing key should just go through that queue. This is fine because there is enough time in between the messages such that I'm able to detect existence of this route.

When two messages with the same routing key X come in within a small time interval (keep in mind this is concurrent behaviour as shown in the diagram) they are both binded to individual queues.
How can I treat this at the RMQ side? I don't want to have to write locking behaviour at the application side. Is there any way for RMQ to declare unique routing keys? I.e. if routing key X is already binded to a queue then subsequent attempts at binding the same routing key to a different queue is not possible.

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/0zWUg1HoXUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0sn22esra8zsfJP7usw48KanVE59p-K2QLw_NibR_UKkA%40mail.gmail.com.

[Michal Kuratczyk]

The simplest solution I can think of is to consistently name the queue. If the queue name matches the routing key, perhaps with some prefix and/or suffix, the second process would simply re-declare the same queue and bindning.

I'd also be curious why you need to declare the queues and bindings dynamically. If you can always find an app to process a certain type of messages, why doesn't that app just declare the queue and binding upfront?

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CACdsx4HAof85Xr9GrXqcLAsAM16p2FZ5mcZDbxd39H7q8SAD8g%40mail.gmail.com.

--

Michał

RabbitMQ team

[Hisham Najem]

I do not know the routing keys upfront. I am binding them to available consumers based on a round robin approach.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0vZeN31U7qMjLdEfzqmruTi-K338PSFp5x2f9NBJUmBuA%40mail.gmail.com.

[Soumen Mukherjee]

Perhaps something wrong in the way it has been coded , may be a thread safety issue ?

Is that something that you can share for peer review...

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CACdsx4FDk7kXYGJ8iqzXsc6r3V7NiAVCMpA_oRBiJm6z3x4r%2BA%40mail.gmail.com.

[Hisham Najem]

Hello Soumen,
I realize it's a thread/parallelism safety issue -- that is why I used the hazelcast lock. I was just wondering if there is a way to let RMQ handle that for me. Perhaps through their own locking mechanism (on the routing key) or through unique routing keys.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAFrNoHRzEvASCn8oztsXwAOoBeySWyCr7sZWKZXYP104Yv2U5A%40mail.gmail.com.

[Hisham Najem]

Basically, I want only one route to ever exist with routing key X.

[Michal Kuratczyk]

Hi,

A direct exchange basically gives you that - as long as the queue name is deterministically derived from the routing key (in the simplest case - it's just equal to the routing key) then this operations is idempotent - it doesn't matter how many apps declare this queue and binding, they will just re-declare the same thing. Then it's a matter of consuming from that queue - if multiple consumers can consume from a single queue you don't have to do anything else, if that's not ok then you can use a single active consumer to make sure only one of the apps consumes from that queue/routing key.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CACdsx4FbHxBeqOxGvtS972CM8BSKOLqkG1uxeLRHeJaBdJnyeQ%40mail.gmail.com.

--

Michał

RabbitMQ team

[Hisham Najem]

Hello,

Thanks for your responses. Bare with me as we try to solve this issue.

The queue name is NOT known ahead of time, multiple distinct routing keys can be bound to it. Basically I want distinct routing keys amongst N queues.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0vL3MMGcz24ZG547KLoy7gwL%2BMYJTk1q8Riz8eznXBTRA%40mail.gmail.com.

[Michal Kuratczyk]

Hi,

This is your current design - I guess a queue per app/instance and you want different routing keys forwarded to different clients. But if you change the logic and have a queue per key and consumers potentially consuming from multiple queues.

Another approach could be to use the Consistent Hashing Exchange: https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_consistent_hash_exchange.

Then you could have each app/instance declare it's queue and the exchange would just uniformly distribute messages between queues and messages with the same routing key would end up in the same queue.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CACdsx4FUOfVsCnM1F9zi%3Di-9GZmFPgOyZL8KpXQe5bToaffx5w%40mail.gmail.com.

--

Michał

RabbitMQ team

[Hisham Najem]

Hello,

The system may have millions of routing keys so the queue per routing key is not a good idea even if it would work. 
The consistent hash exchange topic is very interesting and I will take a look. Thank you for your time. I'll let you know if it solves the issue.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0tUma2EXK%3DMwcUE3tAV3cr0qXx5Df_bXQWySQ9_poz%2BnQ%40mail.gmail.com.