RabbitMQ Langohr no heartbeat

[David Smith]

Can anyone help me with this http://stackoverflow.com/questions/36596446/rabbitmq-langohr-no-heartbeat

I am setting up a connection to Rabbit using

(rmq/connect (assoc (:rabbit conf) :requested-heartbeat 3))

I have 3 subscribers to 3 channels, no producers. When I shutdown the rabbit server, nothing happens to my client. I would expect it to report missing heartbeat after 1.5 seconds and close the connection after 3 seconds. I have left the program running with rabbit down for quite a few minutes and nothing.

I'm using langohr 3.5.1

I have checked .getHeartbeat rabbit-conn and it is correctly set to 3. I have also tried setting :connection-timeout 5 which I believe is 5ms. If I set that to 1 then my app fails to start up (timeout too small to make the initial connection), so I'm sure that setting is correct as well.

[Michael Klishin]

You can see effective heartbeat in the management UI on the connection page.

Heartbeats do not close connection and it't not correct that connection should be considered dead in 1.5 seconds with a 3 second interval: it will be 3 seconds.

I'm curious if Langohr simply tries to re-connect because missed heartbeats are treated the same way as connection loss in recent Java client versions: is connection recovery enabled?

--
You received this message because you are subscribed to the Google Groups "clojure-rabbitmq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-rabbit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Smith]

I have tried (connection (assoc (:rabbit conf) :requested-heartbeat 3 :automatically-recover false :connection-timeout 5)) based on what you said but no joy, I would have thought I would at least see something in the logs to suggest it's trying to do something. I thought the connection was supposed to be closed if 2 heartbeats were missed?

Thanks.

--
You received this message because you are subscribed to a topic in the Google Groups "clojure-rabbitmq" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure-rabbitmq/Z39EavO9Eas/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure-rabbit...@googlegroups.com.

[Michael Klishin]

That is correct but what you provide in the Java (and most other) client is a timeout, not a heartbeat transmission interval.

See the effective heartbeat timeout in the UI, it could be a Clojure client issue.

What OS is this on?

[Michael Klishin]

Also, please post full RabbitMQ logs and ideally a Wireshark traffic capture. With the amount of information we have at the moment the best we can do is guessing.

On 14 abr 2016, at 14:39, David Smith <shmi...@gmail.com> wrote:

[David Smith]

Isn't this a mailing list for the Clojure client?

[Michael Klishin]

It is. What difference does it make? We need to base our assumptions on data and network traffic capture is the best data we can get. Server logs are also helpful and trivial to collect.

[David Smith]

It was just that it sounded like you were answering the question from a Java perspective but didn't know anything about the Clojure wrapper, but I can see from the connections page that you wrote it :) I couldn't see anything about the heartbeat on the connections page, just:

product	Langohr
copyright	Copyright (C) 2011-2014 Michael S. Klishin, Alex Petrov
capabilities	exchange_exchange_bindings: true connection.blocked: true authentication_failure_close: true basic.nack: true publisher_confirms: true consumer_cancel_notify: true
information	See http://clojurerabbitmq.info/
version	3.0.x
platform	Clojure 1.8.0 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_66

[David Smith]

Nothing in the logs, just the open connection line then nothing:

=INFO REPORT==== 14-Apr-2016::14:27:45 ===

accepting AMQP connection <0.1484.0> (192.168.99.1:50383 -> 172.17.0.2:5672)

[Michael Klishin]

Langohr is a wrapper around RabbitMQ Java client and in recent versions relies on it to also perform connection recocovery.

Heartbeats were never implemented in Clojure.

Your Langohr version is from 2014 or so. Have you considered trying the most recent release (or does it still use 2014 in client properties? Also possible)

Anyhow, investigating this kind if issues would be the same with every client library:)

[Michael Klishin]

OK, that also reveals that RabbitMQ is running in a remote host,

which I expected but wasn't sure.

Are there any load balancers involved?

[David Smith]

I'm using 3.5.1 which is the latest version according to github and clojars.

I am running rabbit as a docker container using docker-machine and image rabbitmq:3-management

[David Smith]

What I see in langohr is that the following are set:

(doto cf
  (.setClientProperties   client-properties)
  (.setUsername           username)
  (.setPassword           password)
  (.setVirtualHost        vhost)
  (.setHost               host)
  (.setPort               final-port)
  (.setRequestedHeartbeat requested-heartbeat)
  (.setConnectionTimeout  connection-timeout)
  (.setRequestedChannelMax requested-channel-max))

I am setting connection-timeout and requested-heartbeat, isn't that what you meant by setting a timeout?

[Michael Klishin]

No, I wasn't referring to the TCP connection timeout.

Can you please take a Wireshark capture? Our team is quite busy this and next week.

[David Smith]

Just installing it now

[David Smith]

Attached is a wireshark capture, hope this helps.

[David Smith]

Hi Guys,

I realise you are busy, I'm just checking the status on this to make sure it doesn't get forgotten :)

When I posted the issue initially it was just a theoretical problem but we have now had a situation where it caused real problems.

Thanks,

David.

[Michael Klishin]

I am traveling till mid-May, we will see how it goes.

--

[David Smith]

I have a new wireshark capture (attached) where I can see the heartbeat however there is no reaction from the client when the connection goes down. 

[David Smith]

OK, solved to some extent. When I shutdown my application properly (i.e. using lein trampoline run to start it) after I had shutdown Rabbit I noticed that I got an exception telling me that the rabbit connection had already been shut down. It therefor seems that the heartbeat is working correctly, the only issue is that the exception that the rabbit client must have raised after shutting down the connection is swallowed up somewhere. My rabbit connection is initialised in the main function of my application so I would have expected an exception to be thrown and appear here however there was no indication of anything. I can work around this by adding a ShutdownListener however it doesn't seem right that I set :automatically-recover false and it silently keeps going with a closed rabbit connection?