Hearbeat timeout error while Consumer receiving a messages

[Andrej Shalaev]

Hello all!

I found strange behavior of RabbitMQ broker v3.5.6 while experimenting with consumer.

Situation is follows:

 * Consumer negotiated default heartbeat from the broker as 60sec.
 * Consumer is set to _not to acknowledge_ messages
 * Consumer receiving messages from the topic with period about 20sec

Result (from the log):

=ERROR REPORT==== 20-Mar-2016::06:33:01 ===
closing AMQP connection <0.4275.46> (127.0.0.1:50611 -> 127.0.0.1:5672):
Missed heartbeats from client, timeout: 60s


I've used tcpdump to capture the stream between the broker and consumer and found that broker does not send a heartbeat frames, so there's no cause for consumer to reply on them. As result, the connection is dropped.

* If messages are going slowly than heartbeat timeout, then broker starts sending hearbeat frames and all is ok!
* If Consumer is set to acknowledge messages all is ok too

Looks like bug in the broker. It is?

[Tom Molesworth]

Hi Andrej,

On 25/03/16 15:18, Andrej Shalaev wrote:

Hello all!

I found strange behavior of RabbitMQ broker v3.5.6 while experimenting with consumer.

Situation is follows:

 * Consumer negotiated default heartbeat from the broker as 60sec.
 * Consumer is set to _not to acknowledge_ messages
 * Consumer receiving messages from the topic with period about 20sec

Result (from the log):

=ERROR REPORT==== 20-Mar-2016::06:33:01 ===
closing AMQP connection <0.4275.46> (127.0.0.1:50611 -> 127.0.0.1:5672):
Missed heartbeats from client, timeout: 60s


I've used tcpdump to capture the stream between the broker and consumer and found that broker does not send a heartbeat frames, so there's no cause for consumer to reply on them. As result, the connection is dropped.

That's not how heartbeats work - there needs to be a packet (any packet) on the wire at least once in each heartbeat interval. The heartbeat frames are just a convenient "do-nothing" packet for keeping the connection alive. Neither the client nor the server should "reply" to a heartbeat - they just need to make sure to generate some outgoing traffic before too many heartbeat intervals expire.

The broker error is indicating that the client failed to send any packets. If there's a 60s heartbeat interval then I'd expect to see some traffic from the broker to keep that side of the connection alive - but a lack of messages there would show up as an error in the client logs, not the server.

cheers,

Tom

[Andrej Shalaev]

Hi Tom, and all

пятница, 25 марта 2016 г., 19:07:20 UTC+3 пользователь tom написал:

Hi Andrej,

That's not how heartbeats work - there needs to be a packet (any packet) on the wire at least once in each heartbeat interval. The heartbeat frames are just a convenient "do-nothing" packet for keeping the connection alive. Neither the client nor the server should "reply" to a heartbeat - they just need to make sure to generate some outgoing traffic before too many heartbeat intervals expire.

The broker error is indicating that the client failed to send any packets. If there's a 60s heartbeat interval then I'd expect to see some traffic from the broker to keep that side of the connection alive - but a lack of messages there would show up as an error in the client logs, not the server.

There's no lack of messages from the broker. In the attachment is filtered pcap-dump of the Broker-Consumer session which is dropped by the Broker due to heartbeat timeout.  Please pay your attention at the frequency of messages and lack of heartbeat frames from the both sides.

My main question: what's best solution for resolving such situation?  IMHO, there's little ambiguity in the AMQP specification regarding heartbeat management.

[Tom Molesworth]

Hi Andrej,

On 30/03/16 15:15, Andrej Shalaev wrote:

пятница, 25 марта 2016 г., 19:07:20 UTC+3 пользователь tom написал:

That's not how heartbeats work - there needs to be a packet (any packet) on the wire at least once in each heartbeat interval. The heartbeat frames are just a convenient "do-nothing" packet for keeping the connection alive. Neither the client nor the server should "reply" to a heartbeat - they just need to make sure to generate some outgoing traffic before too many heartbeat intervals expire.

The broker error is indicating that the client failed to send any packets. If there's a 60s heartbeat interval then I'd expect to see some traffic from the broker to keep that side of the connection alive - but a lack of messages there would show up as an error in the client logs, not the server.

There's no lack of messages from the broker. In the attachment is filtered pcap-dump of the Broker-Consumer session which is dropped by the Broker due to heartbeat timeout.  Please pay your attention at the frequency of messages and lack of heartbeat frames from the both sides.

So the broker drops the connection because the client is badly-behaved: it should be sending a packet at least once every heartbeat interval. The broker is doing exactly the right thing by dropping the connection.

The broker would not be expected to send any heartbeat frames: it is already sending traffic, so these are unnecessary. The client is at fault for not implement heartbeats correctly - which client are you using?

My main question: what's best solution for resolving such situation?  IMHO, there's little ambiguity in the AMQP specification regarding heartbeat management.

I agree entirely:

" The peer should make a best effort to send heartbeats at specified intervals. Heartbeats can be sent at any time. Any sent octet is a valid substitute for a heartbeat, thus heartbeats only have to be sent if no non-heartbeat AMQP traffic is sent for longer than one heartbeat interval. If a peer detects no incoming traffic (i.e. received octets) for two heartbeat intervals or longer, it should close the connection without following the Connection.Close/Close-Ok handshaking, and log an error"

Anyway, it sounds like you have found a bug in the client implementation, so it may be worth mentioning the above text in the bug report - it's from section 4.2.7, https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf

best regards,

Tom

[Andrej Shalaev]

Hello,


среда, 30 марта 2016 г., 18:31:54 UTC+3 пользователь tom написал:

There's no lack of messages from the broker. In the attachment is filtered pcap-dump of the Broker-Consumer session which is dropped by the Broker due to heartbeat timeout.  Please pay your attention at the frequency of messages and lack of heartbeat frames from the both sides.

So the broker drops the connection because the client is badly-behaved: it should be sending a packet at least once every heartbeat interval. The broker is doing exactly the right thing by dropping the connection.

The broker would not be expected to send any heartbeat frames: it is already sending traffic, so these are unnecessary. The client is at fault for not implement heartbeats correctly - which client are you using?

I'm using self-written client based on AMQP-CPP library. The problem is that i did not find any public method(s) to send heartbeat explicitly. So there's seems one solution - to acknowledge each message explicitly.

Anyway, thank you for clarifying the situation!