Rabbitmq haproxy timeouts

[Alexey Makarov]

Hello all. There is a problem with haproxy timeouts. I'm using it in front of two rabbitmq nodes. In Haproxy's log i see some termination state flags.
When nagios checking nodes, log looks like this: SD flag
[10/Nov/2014:14:48:01.190] from NAGIOS IP to Rabbitmq:5672 rabbitmq_01 SD req_size=6 resp_size=0

And when client connects, log looks like this: cD flag
[10/Nov/2014:14:43:52.692] from  Client to Rabbitmq:5672 rabbitmq_-02 cD req_size=853 resp_size=686

There are something with timeouts i think. But i dont know how to fix this. Cant understand how to set timeouts, to avoid this problem
From the web UI of rabbit, in "connections" i see timeout "580s"  Client connect from tomcat.
I've tried lots of configurations, Now haproxy conf is:

defaults
        log     global
        mode    tcp
        option  tcplog
        option  dontlognull
        option redispatch
        retries 3
        timeout connect 5000
        timeout client  200s
        timeout server  200s

frontend localnodes
    bind *:5673
    mode tcp
    option tcplog
    log global
    log-format [%t]\ from\ %ci:%cp\ to\ %si:%sp\ %s\ %ts\ req_size=%U\ resp_size=%B
    default_backend rabbit

backend rabbit
        mode tcp
        log global
        option tcplog
        option tcpka
        balance roundrobin
        server rabbitmq_01 :5672 check
        server rabbitmq_02 :5672  check

[Michael Klishin]

If you take a look at RabbitMQ log you'd likely find messages about abruptly
closed connections roughly 200 seconds after the connections were accepted.

What you're seeing in the management UI is RabbitMQ heartbeat timeout: a period
of time after which RabbitMQ and client alike should consider the peer dead. This includes
2 skipped heartbeat frames. In other words, RabbitMQ and a correct client should
send a heartbeat frame roughly every [1/2 the heartbeat timeout] seconds.

That makes it ~ 290 seconds by default (which is your case). Your HAproxy timeouts
are set to 200 seconds. Which means that if there is no other activity on open connections,
HAproxy will consider it stale and close it after 200 seconds.

It's not hard to tell what the solution should be: either decrease heartbeat interval (say, to 10-20
seconds) for your clients or bump HAproxy timeouts to at least 300 seconds.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Alexey Makarov]

Yes you're right, when log level was info, i saw messages about connections closed abruptly. then changed log level to  error. Ok so i need to set all three timeout parameters about to 300s? Or just timeout client and server?

[Michael Klishin]

On 10 November 2014 at 16:13:45, Alexey Makarov (zip...@gmail.com) wrote:
> so i need to set all three timeout parameters about to 300s? Or
> just timeout client and server?

Alexey,

We do not generally do HAproxy support on this list. Since HAproxy will consider
both client and server connections to be stale with your settings, yes, both client
and server inactivity timeouts need to be increased.

[Alexey Makarov]

Hmm now the termination state is SD, not cD.
[10/Nov/2014:15:25:42.576] from  Client to Rabbitmq:5672 rabbitmq_02 SD req_size=937 resp_size=681

[Alexey Makarov]

Ok thanks for that, i'll write to haproxy group.

[Michael Klishin]

 On 10 November 2014 at 16:28:25, Alexey Makarov (zip...@gmail.com) wrote:
> Hmm now the termination state is SD, not cD.

I suspect S means "server" and C means "client, that's all.