Persistent Message are lost after server Restarts

[Rajanbabu Manoharan]

Hi All,

We are using RABBITMQ for sending messages from Client to server.

Further these messages will be processed and stored in Database.

Here we are using following code to persist the messages in the Queue.

var basicProperties = Channel.CreateBasicProperties();
basicProperties.Persistent = true;
Channel.BasicPublish(exchange, routingKey, basicProperties, body);

It was working properly. But today when we reboot the server all the messages(17K messages) stored in the Queue has been cleared.

Any Reason why the messages were not persisted after server reboot?

Is there Any way to retrieve this messages?

In Physical Disc which location these messages will be persisted?

We are using RabbitMQ 3.7.9 version.

Regards,

Rajan

[Luke Bakken]

Hello,

Please be aware that the version of RabbitMQ you are using is very out-of-date and unsupported.

Are the queues to which you are publishing messages created with "durable" set to true?

Can you provide a complete set of steps to reproduce what you observe?

Thanks -

Luke

[Rajanbabu Manoharan]

Hi Luke,

Thanks you for your reply.

The Queue is created with Durable flag set to true. Below is the screenshot

There are 2 nodes (RabbitMQ Servers)are running. Both are processing the queue messages.

When I noticed in the RabbitMQ Portal, below error message was displayed. 

To Resolve this, I Restarted one of the node(Ex : Server1).

After Restart all the messages(17K) are cleared. These are production messages which is lost.

Any Idea why persistent messages were cleared? Is it possible to recover the messages?

Kindly help here.

Thanks,

Rajan

[Luke Bakken]

Hi Rajan -

It's likely you will not be able to recover your messages. Since you are using a two-node cluster (which is NOT recommended), my guess is that by restarting a server you promoted an unsynchronized mirror.

Here is what you should do:

• Upgrade RabbitMQ to the latest version, which is 3.11.13
• Use three nodes, minimum (https://www.rabbitmq.com/clustering.html#node-count)
• Use quorum queues - https://www.rabbitmq.com/quorum-queues.html

Thanks,

Luke

[Rajanbabu Manoharan]

Hi Luke,

Thanks for the information.

So when network partition error happened, How to resolve/recover this error from 2 node server without loosing messages?

Also when network partition error happened, Why consumers are not able to process the messages from Queue?

Thanks,

Rajan

On Wednesday, April 12, 2023 at 11:14:16 PM UTC+8 Luke Bakken wrote: