Rabbitmq client sending [FIN,ACK] To server at idle state

[moshe cohen]

hi,

my program is initiating a connection with rabbitmq.

i use blocking connection with heartbeat of 60.

suddenly, the client is sending FIN,ACK to the rabbitmq server. the server reply with rst,ack and the connection is now closed.

this is happening on idle state, where no data is sending from client to server and no exception has been raised.

my program is still running and no exception err has been recorded in my log file.

i recorded the network packets with wireshark sniffer several times and noticed that a proper heartbeat transactions has been made between my client and the server (heartbeat sent from client to server and from server to client as well).

i need to keep a live the connection for a long time (infinity actually ... ) so every 10 seconds i use process_data_events with sleep of 10 seconds and lock the connection variable resource.

while 1:

     lock.aquirce()

     time.sleep(10)

     self.connection.process_data_events()

     lock.release()

what am i doing wrong ?

thanks,

moshe.

[Emre gündoğdu]

Hi,

I have got the same problem.We are using mqtt plugin and some devices using mqtt client getting sometimes FIN + ACK package from broker nodes, and then RST the TCP connection.Following screenshot show us TCP connection reset occurrence

And I have seen our mqtt client send keep alive  package every 10 second.Our broker configuration using default settings about  theartbeat and keepalive.

Our Broker is consist of three nodes EC2 instance 

Broker nodes behind AWS Classic Load Balancer and our Load Balancer default IDLE timeout is 900 seconds.

Our Cluster diagram as below. Why our mqtt client getting FIN + ACK package form Broker to closing the TCP connection ?

11 Ağustos 2019 Pazar 10:48:40 UTC+3 tarihinde moshe cohen yazdı:

[Luke Bakken]

Hi Moshe,

It looks like you are using Pika. I recommend enabling DEBUG logging to see what is happening:

https://github.com/pika/pika/blob/master/examples/basic_consumer_threaded.py#L10-L14

Thanks -

Luke

[Emre gündoğdu]

Hi Luke

What do you think about our problem that is related to mqtt connection reset as mentioned above.Do you have any troubleshooting to exact connection problem in which component caused by such as broker , Load Balancer , broker nodes or client side ?

If you need to additional wireshark output for the client side i would be attach it ?

Thanks

Emre Gündoğdu

13 Ağustos 2019 Salı 17:26:12 UTC+3 tarihinde Luke Bakken yazdı:

[Luke Bakken]

Hi Emre,

The best place to start are the RabbitMQ logs. What is logged when a connection stops?

Can you reproduce the issue in an environment that does not use a load balancer?

Thanks,

Luke