RabbitMQ block a few connection but the memory is absolutely enough

[Choo Tze Sien]

I have a few MQTT publisher being blocked by RabbitMQ for no reason. The memory of my instance still more than enough and other connection able to run properly.

Is there anyone know how is the blocking logic is implemented?
In the documentation, the only reason connection get blocked is becuase of the low resource threshold is reached. Would like to know if there's any other parameter or policy that will trigger the connection blocking logic.

Thanks!

[Luke Bakken]

Hello,

You haven't provided enough information for us to assist you.

• It would be great to know what version of RabbitMQ and Erlang you're using, as well as on what operating system.
• Please attach your full RabbitMQ configuration to your response.
• What is in the RabbitMQ log files at the time of the blocked connections?
• When you see blocked publishers, what is the output of this command? rabbitmq-diagnostics alarms
• Also, what does the "Overview" screen show in the management interface?

Thanks!

Luke

[Choo Tze Sien]

I am running my rabbitmq with docker, the image is FROM rabbitmq:3.8.16-management

RabbitMQ 3.8.16Erlang 24.0.2

I have attached some screenshot

[Choo Tze Sien]

Will a connection been blocked because of rabbitmq authentication reject too many times?

[Luke Bakken]

Hello,

No, authentication errors should not cause this issue.

RabbitMQ 3.8 is completely out-of-support, please upgrade.

If you are able to reproduce this issue using the latest version of RabbitMQ and Erlang, please let us know.

Thanks,

Luke

[jason chang]

Hi,

I also encountered the same problem.
The RabbitMQ version I am using is 3.11.10, and the Erlang version is 25.0.

Current environment has more than 500 devices connected to RabbitMQ via MQTT.
Each device sends data to RabbitMQ every minute.
Then redirect these data to another queue (such as MyQueue).
Then use another program to go to MyQueue to consume the data.

When the memory and hard disk resource are sufficient, sometimes connection state blocked occurs.
 
When connection state blocked occurs, the following situations will occur:
1. The operation of RabbitMQ management UI becomes very slow.
2. The data in MyQueue keeps accumulating (accumulating to hundreds of thousands of data), but our program just cannot consume the data.
3. A large number of logs appear, including:
   a. "Channel is stopping with 64 pending publisher confirms".
   b. "MQTT: a socket write failed, the socket might already be closed".
   

Erlang's CPU usage is between 5-10% as usual, but about half an hour later the system returned to normal.
Huge number of data can be consume normally, the connection will no longer have a blocked state, and the log will no longer have many of the abnormal logs mentioned above.

It is also worth mentioning that the log usually shows the log of get_disk_info timeout.
Also, the disk space cannot be obtained in the overview, it is all shown as 0, but my system has more than 50GB of free space.

Are there any other recommended commands to observe the current behavior when the above blocked state occurs, or are there any recommended configuration settings to reduce the occurrence of the problem?

Thanks.

[Luke Bakken]

Hello,

RabbitMQ 3.11.x is not eligible for free community support. Please upgrade. In addition, RabbitMQ 3.13.x has major improvements to MQTT that you would probably benefit from.

https://github.com/rabbitmq/rabbitmq-server/blob/main/COMMUNITY_SUPPORT.md