TTL in Quorum Queues

300 views
Skip to first unread message

Shilpa Kumar

unread,
Oct 14, 2020, 6:43:01 AM10/14/20
to rabbitmq-users
Hi 

Can you plesae tell me whether TTL has been implemented in Quorum queues.From old posts i could see that its planning to get implemented .

If not what is the best approach for a replacement of TTL in classic queue to the Quorum queue?

Many Thanks
Shilpa

Karl Nilsson

unread,
Oct 14, 2020, 8:04:48 AM10/14/20
to rabbitm...@googlegroups.com
Hi,

Is this message TTL you are asking about or queue TTL?

Queue TTL for Quorum Queues will be available in the next release (3.8.10).

Message TTLs we are currently considering. My concern around message TTLs are the use cases for them as spontaneously deleting messages that the Quorum Queue has spent a non-trivial amount of work persisting across multiple nodes seems to go against the goal of ensuring message data safety. Can you explain how and why you are using message TTLs?

Cheers
Karl
From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Shilpa Kumar <shilpa....@gmail.com>
Sent: 14 October 2020 11:43 AM
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: [rabbitmq-users] TTL in Quorum Queues
 
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/b8f86dc2-290a-497c-b25b-f9c4c257ac74n%40googlegroups.com.

Shilpa Kumar

unread,
Oct 14, 2020, 8:57:20 AM10/14/20
to rabbitmq-users
Thanks Karl. I meant about Queue TTL. Can you please let me know when 3.8.10 will be released

Many Thanks,
Shilpa

Karl Nilsson

unread,
Oct 15, 2020, 3:57:31 AM10/15/20
to rabbitm...@googlegroups.com
I can't say when the next version will ship - 3.8.9 was only released 21 days ago so it will probably be a few weeks at least.

Out of curiosity - why do you need Queue TTL? (I.e. the queue will be auto deleted after some period of inactivity)
Sent: 14 October 2020 1:57 PM
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: Re: [rabbitmq-users] TTL in Quorum Queues
 

Shilpa Kumar

unread,
Oct 15, 2020, 4:43:54 AM10/15/20
to rabbitmq-users

Hi Karl,

Thanks for the update. 
We are planning to change from classic queue to Quorum Queue. We have a mechanism where we need to retry our dead letter messages after TTL period.
Since TTL is not available in Quorum queues, I am finding a way to replace TTL.
Do we have any other mechanism for replacing this TTL in classic Queue or do i need to wait till its implemented. If so , can you just assure that it will be available within a month or so ?

Many Thanks,
Shilpa

Karl Nilsson

unread,
Oct 15, 2020, 5:53:05 AM10/15/20
to rabbitm...@googlegroups.com
Ok that sounds like Message TTL applied to the queue as a policy, not Queue TTL.

Delayed retries is something that many people want to do - you could use a Quorum Queue as your main queue a DLX configuration where the DLX queue is a non-replicated durable classic queue with a message TTL policy. Assuming the messages don't need be delayed too much this should probably be sufficient for most scenarios.
Sent: 15 October 2020 9:43 AM
To: rabbitmq-users <rabbitm...@googlegroups.com>

Shilpa Kumar

unread,
Oct 15, 2020, 7:22:42 AM10/15/20
to rabbitmq-users
Thanks Karl.

We thought of moving to Quorum queue to avoid the loss of messages from classic queues.So if we are using this DLX as classic and main queue as Quorum , is there any chance of loss if one node is down or something similar? As we highly need Quorum in our application (with this TTL or any alternative to TTL) could you please advise this?

And when the stable version with TTL will be released?

Many Thanks in advance 
Shilpa

Karl Nilsson

unread,
Oct 15, 2020, 8:47:58 AM10/15/20
to rabbitm...@googlegroups.com
I haven't tested this, but you could try to dead letter into the delayed message exchange [1] - currently it stores messages in a node local database but that means that each delayed message is stored on the current node rather than a non-replicated queue which is only stored on one node (which may or may not be available).

Else you could make your retry queue a classic, ha queue but I wouldn't recommend it.

I don't have an ETA for message TTL for quorum queues. There are some inherent technical issues around dead lettering that needs to be reviewed to ensure this is the most appropriate way to achieve certain patterns going forward.
Sent: 15 October 2020 12:22 PM

Shilpa Kumar

unread,
Oct 16, 2020, 4:03:39 AM10/16/20
to rabbitmq-users

Thanks Karl for the detailed explanation ..

Ádám Kiss

unread,
Jan 20, 2021, 6:00:59 AM1/20/21
to rabbitmq-users
Hi All!

Karl mentioned that queue ttl would be added to qourum queues in 3.8.10:
> Queue TTL for Quorum Queues will be available in the next release (3.8.10).

But in the release notes I don't see this change:
https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.10

Did this not make it, or was it just left out in the changelog?

PS: our use cases is the same: delayed retry with dead letter + ttl

Thanks,
Adam

Ádám Kiss

unread,
Jan 20, 2021, 6:07:13 AM1/20/21
to rabbitmq-users
Sorry, I just realized I mixed up queue TTL with message TTL set as queue policy as well. Still, the queue ttl question might be interesting to someone.

Is there any update on solution for delayed redelivery?

Thanks,
Adam

Karl Nilsson

unread,
Jan 20, 2021, 6:07:40 AM1/20/21
to rabbitmq-users
I think it was left out of the release notes. As far as I can see you can now use the x-expires queue argument to set the queue ttl value. Also it seems the docs were not updated either so we'll do that.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/efaec6e1-f3a3-4863-875a-c04a04f5aaf4n%40googlegroups.com.


--
Karl Nilsson

Karl Nilsson

unread,
Jan 20, 2021, 6:13:20 AM1/20/21
to rabbitmq-users
There is no update apart from the suggestion to try using the delayed message exchange instead of a queue with message TTL. Dead-letting is inherently unsafe due to its at-most-once semantics which is why we are hesitant to go down that path going forward. So if the dlx target queue is hosted on another RabbitMQ node from the source then the message has to cross the network which increases the chances of something going wrong. Using the delayed message exchange this is never the case as each node stores its delayed messages locally.


To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/874979bd-093e-414f-85d8-60da3c5eb77en%40googlegroups.com.


--
Karl Nilsson
Reply all
Reply to author
Forward
0 new messages