RabbitMQ channel is dying after idle when using MassTransit from a consumer creating two channels

[Tim Holden]

We have a windows service that consumes messages from a RabbitMQ queue.  The service establishes two channels with RabbitMQ.  One of those channels is used by the service itself when publishing notifications.  The other channel is what we are consuming. 

 

We are using RabbitMQ Client version 3.0, and MassTransit version 2.7.2.  The consumer and the RabbitMQ server are on different machines.

 

What we have discovered is that the channel which we receive messages on is dying.  However, it only happens after that channel has been sitting idle for a while with no messages received.  Usually this means that we get one message every 24-hours.  And, there are no errors being thrown from MassTransit (we use log4net in debug mode) or messages logged from MassTransit to tell us why.  I checked the RabbitMQ log on the RabbitMQ server and it too did not show any issues.

 

My initial thought was that something on the network was killing the channel since it was sitting idle.  However, the notification channel we create as part of the consumer is also idle during the same period of time.  So, if it was the firewall or something doing this then it would have killed both idle channels, not just one.

 

My current theory is that there is some sort of error on the channel when receiving a message that is not getting reported and that it is causing the channel to shut down.  Is there some sort of logic within MassTransit that might have problems with only receiving messages once every 24-hours?  Does anyone have any other ideas?

 

Thanks for your help.

[Tim Holden]

I'm able to reproduce the problem somewhat.  I'm not sure how long I need to let it sit there, but if I let the consumer sit with the channel idle then post a message then I see it waiting in the channel unacked for a few seconds, then the channel dies.

 

I have let it sit for several hours without issue, however in all of the cases I have seen so far where there are problems it has been over 12 hours that the channel had sat idle before a message was posted.

 

There is nothing in the MassTransit log from the consumer application (in debug mode).  This is the only thing in the RabbitMQ log (IP's edited and removed by me):

 

=INFO REPORT==== 26-Jan-2013::12:42:15 ===

accepting AMQP connection <0.12390.0> ([::1]:49423 -> [::1]:5672)

=WARNING REPORT==== 26-Jan-2013::12:42:34 ===

closing AMQP connection <0.11204.0> (SOME_IP:58215 -> SOME_OTHER_IP:5672):

connection_closed_abruptly

[Henrik Feldt]

You may want to activate heartbeats? Perhaps TCP connections that have no data flowing are closed down?

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To post to this group, send email to masstrans...@googlegroups.com.
To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/8McaghjBxyQJ.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tim Holden]

Thanks Henrik.  That is the conclusion I just came to as well.  I was monitoring the TCP/IP activity and discovered that for connections idle over a particular length of time (under 2 hours), when I tried to post a message to the queue it would re-transmit several times before finally disconnecting the connection.

 

So, it appears we are losing the link for these channels due to inactivity.  It appears our firewall may be shutting them down.

 

I thought I had read somewhere that MassTransit did heartbeats on a default interval.  I'll dig into how to adjust or enable them.

[Tim Holden]

I have enabled HealthMonitoring in MT and it is sending heartbeats.  Hopefully this corrects the issue.

[Tim Holden]

UseHealthMonitoring is not what I needed.  That creates more channels and sends the messages through that, for use by the MassTransit Health Service. 

 

What I discovered is that in order to enable heartbeats on the channel used, I need to do this instead:

     sbc.UseRabbitMq(r => r.ConfigureHost(new Uri(this.MQUrl), h => h.SetRequestedHeartbeat(300)));

It appears the default value for heartbeats for MassTransit RabbitMQ Client is 0, so it doesn't send them without changing it (done in RabbitMQ.Cient.Impl.ConnectionBase::StartAndTune).

[Chris Patterson]

Correct, setting the RabbitMQ heartbeat is the right way to handle it.

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To post to this group, send email to masstrans...@googlegroups.com.
To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/WAbLEfHZ7ecJ.

[Tim Holden]

Thanks Chris.  Yes, it worked great. 

On Saturday, January 26, 2013 10:04:11 PM UTC-6, Chris Patterson wrote:

Correct, setting the RabbitMQ heartbeat is the right way to handle it.

On Sat, Jan 26, 2013 at 6:20 PM, Tim Holden <timothy...@gmail.com> wrote:

UseHealthMonitoring is not what I needed.  That creates more channels and sends the messages through that, for use by the MassTransit Health Service. 

 

What I discovered is that in order to enable heartbeats on the channel used, I need to do this instead:

     sbc.UseRabbitMq(r => r.ConfigureHost(new Uri(this.MQUrl), h => h.SetRequestedHeartbeat(300)));

It appears the default value for heartbeats for MassTransit RabbitMQ Client is 0, so it doesn't send them without changing it (done in RabbitMQ.Cient.Impl.ConnectionBase::StartAndTune).

On Saturday, January 26, 2013 5:54:20 PM UTC-6, Tim Holden wrote:

I have enabled HealthMonitoring in MT and it is sending heartbeats.  Hopefully this corrects the issue.

On Saturday, January 26, 2013 4:53:32 PM UTC-6, Tim Holden wrote:

I thought I had read somewhere that MassTransit did heartbeats on a default interval.  I'll dig into how to adjust or enable them.

On Saturday, January 26, 2013 1:26:53 PM UTC-6, Henrik wrote:

You may want to activate heartbeats? Perhaps TCP connections that have no data flowing are closed down?

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To post to this group, send email to masstrans...@googlegroups.com.

To unsubscribe from this group, send email to masstransit-discuss+unsub...@googlegroups.com.