"Empty response" sometimes - RabbitMQ behind HAProxy

[Alexander Birkner]

Hello,

we've switched our RabbitMQ cluster to HAProxy for load balancing. Since this change we have sometimes a really strange behavior.

I'm using the Ruby Bunny Library and I get randomly the Exception Bunny::NetworkFailure on some servers.

Exception string: "detected a network failure: Empty response received from the server."

I've seen this post already on Serverfault  but it not seems to help me in this case.

This is my HAProxy configuration file:

global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        maxconn 8192
        user haproxy
        group haproxy
        daemon


defaults
        maxconn 8192
        timeout connect 5000
        timeout client  3h
        timeout server  3h


listen rabbitmq 0.0.0.0:5672
        mode tcp
        log global
        retries 3
        timeout connect 5000
        timeout client  3h
        timeout server  3h
        option          clitcpka
        option tcplog
        option persist
        balance leastconn
        server bunny01 bunny01.XXXXXX.net:5672 check
        server bunny02 bunny02.XXXXXX.net:5672 check
        server bunny03 bunny03.XXXXXX.net:5672 check

listen stats :1936
      mode http
      stats enable
      stats hide-version
      stats realm Haproxy\ Statistics
      stats uri /
      stats auth XXXXXX:XXXXXXXX

cat /proc/sys/net/ipv4/tcp_keepalive_time
7200

Can you see some mistakes in my configuration file? Or have you another idea? 

Best regards,

Alexander Birkner

[Michael Klishin]

On 17 September 2015 at 11:44:49, Alexander Birkner (alex.b...@gmail.com) wrote:
> I'm using the Ruby Bunny Library and I get randomly the Exception
> Bunny::NetworkFailure on some servers.
> Exception string: "detected a network failure: Empty response
> received from the server."

Have you inspected HAproxy and RabbitMQ logs? 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Alexander Birkner]

Hello Michael,

yes I already did. The RabbitMQ log just reports "connection_closed_abruptly":

=WARNING REPORT==== 17-Sep-2015::12:05:24 ===

closing AMQP connection <0.13773.1701> (194.169.211.XXX:46863 -> 5.101.164.XXX:5672):

connection_closed_abruptly

=WARNING REPORT==== 17-Sep-2015::12:05:24 ===

closing AMQP connection <0.13785.1701> (194.169.211.XXX:46876 -> 5.101.164.XXX:5672):

connection_closed_abruptly

=WARNING REPORT==== 17-Sep-2015::12:05:24 ===

closing AMQP connection <0.21702.1701> (194.169.211.XXX:47317 -> 5.101.164.XXX:5672):

connection_closed_abruptly

=WARNING REPORT==== 17-Sep-2015::12:05:24 ===

closing AMQP connection <0.7409.1663> (194.169.211.XXX:47382 -> 5.101.164.XXX:5672):

connection_closed_abruptly

=WARNING REPORT==== 17-Sep-2015::12:05:24 ===

closing AMQP connection <0.12254.1695> (194.169.211.XXX:47402 -> 5.101.164.XXX:5672):

connection_closed_abruptly

=WARNING REPORT==== 17-Sep-2015::12:05:24 ===

closing AMQP connection <0.22947.1701> (194.169.211.XXX:47782 -> 5.101.164.XXX:5672):

connection_closed_abruptly

The HAProxy log seems not very helpful because it only logs the IP and port of the RabbitMQ client and not the local HAProxy port to the RabbitMQ server to easy find the same connection log entry.

Best regards,

Alexander

[Michael Klishin]

These entries mean that a client (so, HAproxy) has closed its TCP connection unexpectedly.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.