[Bunny] Getting exceptions after a connection is closed due to a timeout. Any ideas?

[Maximilian Tagher]

Hi. My company Heyzap recently start using Bunny to connect to RabbitMQ from our Rails servers (using Unicorn/Resque). We've been using RabbitMQ with our java services for awhile without any issues, but Ruby-side we've regularly been getting this exception:

Trying to send frame through a closed connection. Frame is #<AMQ::Protocol::MethodFrame:0x00000016886ce0 @payload="\x00<\x00(\x00\x00\btimecube\ftimecube.sdk\x00", @channel=1>, method class is AMQ::Protocol::Basic::Publish

(stacktrace available as a gist)

These errors are proceeded by a few timeout errors, which I think is the root cause:

[GEM_ROOT]/gems/bunny-1.7.0/lib/bunny/concurrent/continuation_queue.rb:25:in `pop'

[GEM_ROOT]/gems/bunny-1.7.0/lib/bunny/concurrent/continuation_queue.rb:25:in `block in poll'

/usr/lib/ruby/2.1.0/timeout.rb:91:in `block in timeout'

/usr/lib/ruby/2.1.0/timeout.rb:101:in `call'

/usr/lib/ruby/2.1.0/timeout.rb:101:in `timeout'

[GEM_ROOT]/gems/bunny-1.7.0/lib/bunny/concurrent/continuation_queue.rb:24:in `poll'

(full stacktrace available as a gist)

This happened without a heartbeat (the Bunny default) and with a heartbeat of 300 seconds (the server's value is the default of 580). When the exceptions start, I need to restart the unicorns to resolve the issue.  

I think that some code may be blocking Ruby and contributing to the timeout errors. Notably, these errors would instantly get triggered when a colleague's code started several threads, then called "join" on them while they uploaded files to S3.

The rabbitMQ logs are filled accepting AMQP connection and closing AMQP connection (connection_closed_abruptly) (sample of the logs). These messages happen all the time, so I'm not sure they're related to the exceptions we're seeing.

Any advice on what might be causing the exceptions, and how to fix it? Should we increase the continuation_timeout from 4000 ms? Any ideas on why these exceptions don't seem to go away after they start (which I presume they would if we were able to reconnect to RabbitMQ)?

Version Info:

Ruby: 2.1.2

Bunny: 1.7.0

amq-protocol: 1.9.2

Rails: 4.1.8

Ubuntu 12

RabbitMQ: 3.4.3

rabbitmqctl status

[Michael Klishin]

 On 28 February 2015 at 00:03:35, Maximilian Tagher (feedbac...@gmail.com) wrote:
> Any advice on what might be causing the exceptions, and how to
> fix it?

Connection to your server is lost. It's your responsibility to make sure publishers
can cope with that (automatic recovery can take a while).

Bunny does not use an event loop library, it has its own I/O loop thread
and it would take some deliberate effort to block it.
--
@michaelklishin, github.com/michaelklishin

[Michael Klishin]

 On 28 February 2015 at 00:07:03, Michael Klishin (michael....@gmail.com) wrote:
> Connection to your server is lost.

… and if RabbitMQ logs about abruptly closed connections, it means that client's TCP connection
was closed unexpectedly.

Do a traffic capture with tcpdump or Wireshark and see what may be going on. It can be some kind of
intermediary closing inactive connections, e.g. F5 load balancers consider no activity of 5 minutes
as a reason to close connection. Having a heartbeat timeout in the 6-30 seconds range should
help with that as heartbeat frames will be sent every 1/2 the timeout interval.

http://rabbitmq.com/heartbeats.html
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Maximilian Tagher]

Thanks I'll try that out