How to handle the RabbitMQ message broker being down

[Charles M]

Hi,

I am new to Mass Transit and trying to work out how best to implement it. At the moment I have a simple client/server set up on my development machine with a .Net client and server and also RabbitMQ running locally.

I am trying to work out what should happen when the RabbitMQ broker service is stopped or unavailable (service stopped, machine not accessible, network issues etc).

Reading the MT documentation...

http://masstransit.readthedocs.org/en/latest/overview/valueadd.html

Exception Management
If your connection to the message broker or queue server goes down, MassTransit takes care of trying to reconnect and deal with those failures, so that you don’t have to.

I can see from debugging my C# code that MT will retry to send the message which results in a InvalidConnectionException being thrown.

I suppose I thought (possibly incorrectly), that MT would persist the messages locally and when the RabbitMQ broker was available at some point later it would start to resend the messages.

Or do I need to handle the InvalidConnectionException and put the messages somewhere myself and re-send them later on?

My code to publish the message looks like this...

                Bus.Instance.Publish(

                    new YourMessage() { Text = text },

                    x => { 

                        x.SetDeliveryMode(MassTransit.DeliveryMode.Persistent);

                    }

                );

... and I thought the Persistent delivery mode would store the messages for sending later on but it looks like I am confused as to how this should work.

Thanks in advance for any advice received.

Charles

[Dru Sellers]

"that MT would persist the messages locally and when the RabbitMQ broker was available at some point later it would start to resend the messages." 

It does not do that. :)

i'll have to let Chris respond about the persistent mode.

-d

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/6122d6d8-9e72-418d-991e-296bb93d8a0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Patterson]

There is no local recovery option, if the broker is unavailable, it will retry connections until it is available.

If you can't get an HA cluster setup and reliable, you could run a local broker and use shovel or federation to move the messages from local to the central node when connectivity is available. Of course, this only really works for sending messages, not receiving them.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHwtU2%3DuDnjDfC%3Dtsv88DVZogh0%2BNQb_YGP23RzeNin2PqyZPQ%40mail.gmail.com.

[Charles M]

Hi,

Thanks for the reply.

My application will be collecting data from real time sources and pushing it onto the service bus for processing. 

1. If the message is sent from a web application running in IIS, I presume the message is kept in memory until a time when it can be sent. I'm guessing if there was an App. Pool recycle (or some other reason the web application restarted) then the message could be lost and would never make it to the broker when it became available again?

2. Also, when I am debugging my code I have noticed that there is quite a delay / timeout when attempting to communicate with an unavailable broker. This is causing my code to hang whilst this takes place. Is there anyway to set a timeout so that code execution will move to the exception handler more quickly? I don't want the application to hang if the broker is not available, rather record the error and then move on to the next message.

3. One the exception has been caught, does the message still stay in the buffer for sending when the broker is available again? That is - is the exception just an indicator that something has happened but the message will be sent at a later time without further intervention from my code?

Thanks again.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/6122d6d8-9e72-418d-991e-296bb93d8a0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

[Chris Patterson]

There is no async support like that in MT2, there is in MT3. 

You could always just Task.Run(() => your publish code here) if you don't want to wait, but an AppDomain recycle will lose unfinished tasks.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/9c1b5e6a-cd89-4285-b6c1-f55d1a8c9544%40googlegroups.com.

[David Prothero]

I like installing RabbitMQ on each web server and joining them to a cluster. The RabbitMQ is local to the web server, but because they are in a cluster, all the messages get routed for you automagically. See this article...

http://looselycoupledlabs.com/2014/07/creating-a-rabbitmq-cluster-for-use-with-masstransit/

If you have hundreds of web servers , that might be too big of a cluster and you may want to look into shovels or federation as Chris suggests.

David

[Charles M]

Is there a way to set a limit on the number or retries?

Also, if I publish 1000 messages to the bus does that mean I have 1000 threads attempting to publish the message, or does the bus try to publish 1000 messages next time it detects the broker is available.

Thanks again.

On Tuesday, 6 January 2015 18:14:50 UTC, Chris Patterson wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/6122d6d8-9e72-418d-991e-296bb93d8a0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

[Chris Patterson]

Well, if you're using the TPL it's unlikely that all of those would be scheduled at the same time, so it would be more like 1000 tasks waiting to execute and maybe a few trying to send at a time, but they would all be blocked around the connection to the broker.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/3481d664-f4e2-4f6d-9de7-7cbfd7f12043%40googlegroups.com.