When to use durable and non durable queues?

[Amit Khosla]

Hi experts, 

In our use case, we want ensurity that messages are not lost. 

For this, we have three node cluster with ha-all policy. 

We are using durable queues. 

For partition handling, pause minority. 

As I understand, durability of queues helps to solve cases where when node restarts, it can pick the messages persisted in mnesia. 

Please correct me if I am wrong. 

Also, in 3 node cluster, if we have any node restart, the node will not start with messages but sync from current master node. 

So is there any use case where durable queues solve the problem in a cluster? 

I guess it can solve case where we restart all node at the same time. 

Is there any other use case? 

Please guide. 

Thanks and Regards

Amit

[Michal Kuratczyk]

Hi,

1. Mirrored queues are deprecated - please use quorum queues (they are always durable and automatically replicated if you have a cluster - no need to for a policy)

2. Messages are not stored in Mnesia - they are just stored on disk (Mnesia contains metadata about your queues and other entities)

2. The main use case for both mirrored queues and quorum queues is the same as for any data replication - to have a copy of the data. Without replication, you'd be prone to losing all data or access to all data, if one node failed.

3. If the queue leader fails, one of the followers (mirrors) takes over the responsibility so you don't lose access to your data.

4. A new/restarted node will sync data from the current leader. In case of mirrored queues it needs to sync all data. In case of quorum queues, it only needs to sync the delta between what it already has and what the leader has.

You can summarize the purpose of queue mirroring as this: assuming a well configured cluster and well-behaving applications, you can lose a minority of the RabbitMQ nodes without incurring downtime. For a 3-node cluster, that means you can lose 1 node. In a 5-node cluster, you can lose 2. Quorum queues automatically replicate to 3 nodes (1 leader + 2 followers), even if more nodes are present in the cluster. Replicating to more nodes would give more safety but it's not for free (there is an overhead of replication).

https://www.rabbitmq.com/ha.html

https://www.rabbitmq.com/quorum-queues.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CADGMEYX6gWThY0_wU5zX3sTJ6V-Mf9-6rJFAGaSXC%3DQ3%2BKTOTg%40mail.gmail.com.

--

Michał

RabbitMQ team

[Amit Khosla]

Thanks Michal for the reply!!

My question was related to persisting messages (durability of queue).

I understand by mirroring, we are gaining advantage that if one node crash then other can be elected as master and work keeps going.

But in such scenario, how the durability of queue plays role is my question.

Currently we are on rabbitmq version 3.6.10. The upgrade might take some time to be released to production. So, we are not having option of quorum queues at the moment.

Can you please help me understand the use cases where persisting messages to disk is beneficial in case of mirror queues?

Thanks & Regards

Amit

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0tPgq6sDL4fX9_Ej2eZ%3Ds7VwW%2Ba1b3Y8xguDS%3DoO5kOGA%40mail.gmail.com.

--

Thanks & Regards
Amit Khosla

[Michal Kuratczyk]

Hi,

Well, I guess I don't have to tell you that you are long overdue for an upgrade. :)

Nothing special about persistence in RabbitMQ I guess - just like with anything else, it's there to make data survive restarts and similar events. Yes, an in-memory replicated queue should generally be able to survive most events, but a loss of all nodes would wipe out all data.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CADGMEYU3gPX4jnpNmH7ud2OC%3DNfaJG5NPTtWoScb3Rbzxo4CHQ%40mail.gmail.com.

--

Michał

RabbitMQ team

[Amit Khosla]

Thanks Michal!

In case of all nodes down, will it revive the messages or for some queues, we might loose messages?

If a new node is starting, as you mentioned above will sync from master. What will be behavior of the new node if all cluster was down? At this stage, will it read back from the disk as no one else present in cluster at the given time?

Thanks & Regards

Amit

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0sMn1ESCT6Sxu9P7hSs5tkUnKi8VQhEvhMaBdSdoBjZHg%40mail.gmail.com.

[Michal Kuratczyk]

Whatever was written to the disk, should be restored.

Each node stores the list of running nodes on disk so that if it goes down, it knows what other nodes were still running at the time. When it restarts, it will wait for them to become available, since they could have a more recent state of the cluster and will synchronize the state from them once they are up.

More info:

https://www.rabbitmq.com/clustering.html

I'd recommend experimenting if you want to learn more. There are many scenarios and some details differ from version to version, based on the queue type/configuration, etc.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CADGMEYXdOAvhb6F62yCi8%3DgKk5a0krGCKFr6%3DqG5WLcwr_J_sw%40mail.gmail.com.

--

Michał

RabbitMQ team