Unable to stop messages from piling up in msg_store_persistent

[Shraddha gulur sudheendra]

Hi,

We are using Rabbitmq.Masstransit in our application to consume the message from rabbitmq. I am receiving the acknowledgement (shown as green line) in graph. And there are no messages left in the queue as well. I have also set up TTL for the messages. So after 2 mins it gets removed from the queue. I can see that a message is getting entered whenever a message gets delivered, published and acknowledged. So totally 3 messages. But, even after successfully receiving the acknowledgement, the entries are not getting removed from the .rdq files in msg_store_persistent and hence the folder size is increasing rapidly in the production environment.  What are we missing? How can we fix this issue without having to manually delete the rdq files?

We are using 

RabbitMQ v3.8.2 with Erlang v22.1 and massTransit v5.5.6.0

[ch...@84codes.com]

We at CloudAMQP have seen a few instances of this issue recently in our fleet of servers.  There is a discussion in the github repo here:  https://github.com/rabbitmq/rabbitmq-server/discussions/10329

A temporary solution that has helped in one case is to increase the value of "queue_index_embed_msgs_below" so that more messages are kept in the queue index rather than being sent to the message store.  But this is not advisable if you have large numbers of unacked messages at any given time; that can lead to excessive memory consumption.

Regards,

Chad K

CloudAMQP

[Shraddha gulur sudheendra]

Hi Chad K, 

Thank you for your suggestion. We did try to implement the suggested solution. But before that we were trying to play around with bulk messages. and we noticed the following behaviour. 

When we sent bulk messages in our system, we noticed that once the rdq file reaches 16mb, the messages if consumed are getting deleted. So ideally once the size of rdq file reaches 16mb, the file gets deleted and a new rdq file is created. But this was not observed in the customer's system. Rdq file of 16mb kept on piling up leading to disk space issue. We inspected the DataServer logs shared by the customer and noticed the following error,

2023-11-27 04:46:24.1788||ERROR|Automated Backup process failed to perform full backup: System.Data.SqlClient.SqlException (0x80131904): Cannot open backup device '\\abc\xyz\AutomatedBackup2023-11-27.bak'. Operating system error 5(Zugriff verweigert).
BACKUP DATABASE is terminating abnormally.

Could this error be anyway related to mnesia db not able to delete the rdq files?

Thanks and regards,

Shraddha G S