Getting sporadic AMQ::Protocol::EmptyResponseError in my heardbeat threads

[matob...@gmail.com]

So we recently moved our RabbitMQ cluster behind a NLB in K8S. (Now running RabbitMQ 3.8.9) Since we did that, I am getting these occasional errors in all my clients.

"Exception in the reader loop: AMQ::Protocol::EmptyResponseError: Empty response received from the server."

It retries and then works just fine, so its mostly noise right now.

Versions of bunny we are using is mixed a bit(lots of client apps out there) But I am seeing this on a 2.17.0  bunny version. I'm leaning our our network/infrastructure team as I think its a problem on that side, not our Ruby bunny clients.

So, I guess what I am hoping for is an idea of really what to look at.

I've dug down pretty deep in to the bunny/amp-protocol code. And from what I can tell, it is doing what it should be, it is getting the empty response and then it basically drops its connection and retries.

The RabbitMQ cluster and its NLB are much more of a mystery to me. Any one have ideas of why I would be getting that "empty response from server" message.

[matob...@gmail.com]

Oh one more thing I could add, is that the only thing of note in the RabbitMQ server log as that when my clients decide to retry the connection after said error, I see RabbitMQ log "client unexpectedly closed TCP connection" which I assume is my client, shutting its connection down, as right after that I see my client connect again. log times all match up.

And as it should be important, these are the params I give to Bunny.new(params)

{
  :host=>"somehost.somewhere",
  :port=>30010,
  :username=>"ausername",
  :password=>"password.meow",
  :vhost=>"/somevhost",
  :heartbeat=>60,
  :conn_per_thread=>true,
  :conn_retry_attempts=>5,
  :verify_peer=>true,
  :ssl=>true
}

[matob...@gmail.com]

So yesterday we tried backing out RabbitMQ server to 3.8.5, as the errors seemed to start when we switched to 3.8.9(but other things did change as well, ELB->NLB, new K8S cluster) So I thought I would just change ONE thing at a time, The RabbitMQ version was the easiest as that is under my teams control.

It didn't really help the issue. But I think it at least helps me rule out RabbitMQ server as the issue.  Nothing on any of our clients side changed. So I'm going to try work with the network folks more to try capture some data.

Just thought I would update what I have found so far.

[Michael Klishin]

Nothing around networking, listeners or connection state handling has changed in RabbitMQ 3.8.

The most useful source of information will be server logs and a traffic capture.

> --
> Bunny: http://rubybunny.info
> March Hare: http://rubymarchhare.info
>
> IRC: #rabbitmq on irc.freenode.net
>
> Post to the group: ruby...@googlegroups.com | unsubscribe: ruby-amqp+...@googlegroups.com
> ---
> You received this message because you are subscribed to the Google Groups "Ruby RabbitMQ libraries" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ruby-amqp+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-amqp/5c7b129e-915c-4607-9d2a-05e946578576n%40googlegroups.com.