Shovel delete queues after loose connection Cloud x OnPremise

[Giscard Fernandes Faria]

Hi folks,

I have been working in a project, where we are migrating our monolith/on-premise system to a micro-service architecture. 

We've decided some services will be totally redesigned in order to be multitenant and reach a good performance, those we are moving to the cloud, some services are not that critical, those we are just refactoring and keeping them on-premise due to legacy database dependencies.

After all, we are ending with some services on the cloud and some on-premise, and those have to communicate to each other. In order to achieve that, we've created two cluster of rabbitmq (one on the cloud, the other on-premise), and for connect both of them a third RabbitMQ configured as a Shovel (attached figure illustrated the architecture).

We are using a Dynamic Shovel with exchange to exchange connection in order to get the work done. Basically we configure one shovel for each service (IMPORTANT: we release new services every month):

• SERVICE B: FROM uri=amqp//user:password@cloud-ip/ exchange=boundary routing-key=mycompany.service-b TO uri=amqp//user:password@on-premise-ip/ exchange=boundary routing-key=
  
• SERVICE A: FROM uri=amqp//user:password@on-premise-ip/ exchange=boundary routing-key=mycompany.servoce-a TO uri=amqp//user:password@cloud-ip/ exchange=boundary routing-key=
  

In such a way, if Service B on-premise send a message to RK=mycompany.service-a, the shovel grabs it and send to the cluster on the cloud

In such a way, if Service A on the cloud send a message to RK=mycompany.service-b the shovel grabs it and send to the cluster on-premise

Everything works pretty well, except when the shovel looses connection, once the Dynamic Shovel bound to a exchange creating a exclusive queue, once I loose the connection the queue is deleted and all messages goes down with them.

The problem is that I cannot loose those messages, once they are business events that shall reach its service destination.

One way to solve that is bind the shovel to queues and not exchange, however it does require to local configure a bind from the exchange for each new service that is added.

If I could keep all shovel configuration on the cloud, that would make the shovel administration much easier.

Do your guys know a good way to solve that? I was thinking about a shovel (dynamic + exchange) that do not create exclusive queue and bind always to the same queue generated name, does such a feature exists?

Thanks and Regards.

[Michael Klishin]

Shovel only deletes the queues that are internal to its operation. If you use manual acknowledgements

then all consumed but unprocessed messages from upstream will be requeued because everything

related to connection link, including its channel, will be restarted.

In particular, Shovel uses server-named exclusive queues when you "consume from" an exchange.

That is an edge case and I don't see how Shovel could do it differently given what it has to do

in that case. You don't have to

do that, though: set up routing upstream however you need and consume from a queue

while paying attention to the ack-mode used http://www.rabbitmq.com/shovel-dynamic.html.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Giscard Fernandes Faria]

Thank you Michael.

Regards.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/YBEKppQfUCE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.