MT3 subscriber not auto recovering after temporary RabbitMQ outtage

[Scott McFadden]

From time to time connectivity to our RabbitMQ server may go down.  Once this happens our MT3/Rabbit subscriber no longer receives messages.   The subscriber is a TopShelf Widnows Service.  The publisher is an ASP.NET MVC web app.  The only way I can get the MT subscriber to start receiving messages again is by manually restarting the subscriber / windows service.  

Looks like RabbitMQ-Client added auto recovery in 3.4.0 : http://www.rabbitmq.com/changelog.html 

How do I make it so that the MT3 subscriber auto recovers after short RabbitMQ outtage?

thanks

MassTransit.RabbitMQ 3.0.12-beta

RabbitMQ 3.5.4 server running on Windows 2008 server

scott

[Chris Patterson]

I need to test further, but my reconnection tests were working fine in earlier beta versions and I don't think I've changed anything here.

I'll try to reproduce under load, but I've seen it work so I'm curious what is happening in this situation.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/76470bc0-1e1c-405e-a5d1-9f52bf1a3948%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Scott McFadden]

Chris, In looking at MassTransit::RabbitMqTransport::RabbitMqAddressExtensions.GetConnectionFactory, I noticed the following code:

 var factory = new ConnectionFactory

            {

                AutomaticRecoveryEnabled = false,

                NetworkRecoveryInterval = TimeSpan.FromSeconds(1),

                TopologyRecoveryEnabled = false,

                HostName = settings.Host,

                Port = settings.Port,

                VirtualHost = settings.VirtualHost ?? "/",

                RequestedHeartbeat = settings.Heartbeat

            };

Doesn't AutomaticRecoveryEnabled need to be set to true for the auto reconnect logic to work in RabbitMQ-Client?

On Thursday, September 24, 2015 at 5:01:38 PM UTC-5, Chris Patterson wrote:

I need to test further, but my reconnection tests were working fine in earlier beta versions and I don't think I've changed anything here.

I'll try to reproduce under load, but I've seen it work so I'm curious what is happening in this situation.

On Thu, Sep 24, 2015 at 1:00 PM, Scott McFadden <scott.kenda...@gmail.com> wrote:

From time to time connectivity to our RabbitMQ server may go down.  Once this happens our MT3/Rabbit subscriber no longer receives messages.   The subscriber is a TopShelf Widnows Service.  The publisher is an ASP.NET MVC web app.  The only way I can get the MT subscriber to start receiving messages again is by manually restarting the subscriber / windows service.  

Looks like RabbitMQ-Client added auto recovery in 3.4.0 : http://www.rabbitmq.com/changelog.html 

How do I make it so that the MT3 subscriber auto recovers after short RabbitMQ outtage?

thanks

MassTransit.RabbitMQ 3.0.12-beta

RabbitMQ 3.5.4 server running on Windows 2008 server

scott

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

[Chris Patterson]

MassTransit has its own reconnection logic. The failures that happen with their recovery turned on are pretty strange. Maybe they fixed them but they used to blow up RAM really badly. 

If MT gets disconnected it should reconnect but there may be some other issue at play. 

____
Chris Patterson

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/289fa26b-17c5-4535-963b-2acf676e629f%40googlegroups.com.

[Scott McFadden]

Chris, here is our bus startup code.  Is that good enough for auto recovery support?

m_bus = Bus.Factory.CreateUsingRabbitMq(sbc =>

            {

                var host = sbc.Host(new Uri(setting.ServiceBusHost), h =>

                                                                    {

                                                                        h.Username(setting.ServiceBusUser);

                                                                        h.Password(setting.ServiceBusPassword);

                                                                    });

                

                sbc.ReceiveEndpoint( host, setting.ServiceBusQueue, endpoint => endpoint.LoadFrom(m_container));  //

                sbc.UseConcurrencyLimit(setting.ConcurrentConsumerLimit);

                sbc.UseTransaction(x => x.Timeout = TimeSpan.FromMinutes(setting.TransactionTimeOutInMinutes));

                

                sbc.UseCtMiddleware(m_container.Kernel);                

               

                

            });

[Chris Patterson]

That should be all you need. Can you turn on debug logging and capture some traces of the connection events?

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/76470bc0-1e1c-405e-a5d1-9f52bf1a3948%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/289fa26b-17c5-4535-963b-2acf676e629f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/0d1f52de-97c5-4744-ba62-f1b90bfc5ab2%40googlegroups.com.

[Scott McFadden]

I wonder if our Rabbit server is old enough so that MT auto reconnect logic does not work:

RabbitMQ.Client 3.5.4

Rabbit Server 3.3.0

Erlang 17

Does MT3 require a minimum version of RabbitMQ Server for auto reconnect to work?

thanks

scott

--
You received this message because you are subscribed to a topic in the Google Groups "masstransit-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/masstransit-discuss/z3IkTG7zmnw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHX4WYcHan3AbT8pU0vau6RKaG7ci3SF-1KH1rRGe%2B2t2%3DRPfg%40mail.gmail.com.

[Chris Patterson]

That's a pretty old version, but I can't be sure that some internal behavior changed. I know there were some major changes with 3.4, and then minor ones with 3.5.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAJGQrQCfkgbh4H2ARRwVXwRbSfkDCyvZhxQbh2oLH3zcHi1qOA%40mail.gmail.com.

[Scott McFadden]

Is it ok to mix/match RabbitMQ server versions with RabbitMQ.Client versions?  For example, RabbitMQ server 3.56 (latest) with RabbitMQ.Client 3.54?  Or should you upgrade RabbitMQ.Client at same time of Rabbit server upgrade?

thanks

scott

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHX4WYdijA-5Pp6UfOTfL2ksXBjFD3FtUwZ9t%2Bh4LJ88GZBdhQ%40mail.gmail.com.

[Chris Patterson]

I would not upgrade the client until MassTransit updates the client. RabbitMQ has a habit of breaking interfaces between versions.

The server version typically does not matter, but 3.30 is 1.5 years old.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAJGQrQCAvTMMacqxOvnX64%3DbJTvXpbjLJE1ucOsx0rBr3KWhEA%40mail.gmail.com.

[Alex Michel]

I had similar RabbitMQ reconnect issue with subscribers in MT 2.9. Upgrading MT + MT Rabbit to 2.10 with all dependencies solved it.