Policy with message-ttl not expiring messages

[alam86]

I have a policy that says all messages in all queues should have a TTL of 60 seconds (policy set to 60000). I see the 'TTL' string next to each queue in the admin page (under Features), which indicates that the policy is being recognized. Messages don't expire in these queues though. They just sit there even after 60 seconds. Am I doing something wrong? 

Not sure if it matters, but this is part of a Federated setup, with 2 upstreams and 1 downstream. This policy exists on the downstream broker.

[Michael Klishin]

Hover over the TTL value and see what it is.

Only one policy applies at a time. There can be a different policy in effect.

On Tue, Feb 7, 2017 at 9:52 AM, alam86 <plal...@gmail.com> wrote:

I have a policy that says all messages in all queues should have a TTL of 60 seconds (policy set to 60000). I see the 'TTL' string next to each queue in the admin page (under Features), which indicates that the policy is being recognized. Messages don't expire in these queues though. They just sit there even after 60 seconds. Am I doing something wrong? 

Not sure if it matters, but this is part of a Federated setup, with 2 upstreams and 1 downstream. This policy exists on the downstream broker.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[alam86]

The name of the policy is 'TTL'. Under 'Features' on the 'Queues' page, I see 'D' and 'TTL'. When I hover over 'TTL', it shows 'Policy: TTL'. Here is my policy information:

Name: TTL

Apply To: queues

Pattern: .*

Definition: message-ttl: 60000

Priority: 0

I do have another policy defined for federation, which applies to just 'exchanges'.

On Tuesday, February 7, 2017 at 12:28:23 AM UTC-8, Michael Klishin wrote:

Hover over the TTL value and see what it is.

Only one policy applies at a time. There can be a different policy in effect.

On Tue, Feb 7, 2017 at 9:52 AM, alam86 <plal...@gmail.com> wrote:

I have a policy that says all messages in all queues should have a TTL of 60 seconds (policy set to 60000). I see the 'TTL' string next to each queue in the admin page (under Features), which indicates that the policy is being recognized. Messages don't expire in these queues though. They just sit there even after 60 seconds. Am I doing something wrong? 

Not sure if it matters, but this is part of a Federated setup, with 2 upstreams and 1 downstream. This policy exists on the downstream broker.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Add a consumer to your queue. Messages will then "reach the head of the queue" as mentioned

in http://www.rabbitmq.com/ttl.html#per-message-ttl-caveats and expire.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

The above is not to say that queues can't not have consumers online some of the time

but specifically for per-queue message TTL, and especially when added retroactively, it has an effect.

This limitation does not apply to per-message TTL or queue TTL as far as I can see.

[Parvez Lalani]

Doesn't the documentation indicate otherwise? The 'Caveats' section says: 

"While consumers never see expired messages, only when expired messages reach the head of a queue will they actually be discarded (or dead-lettered). When setting a per-queue TTL this is not a problem, since expired messages are always at the head of the queue. When setting per-message TTL however, expired messages can queue up behind non-expired ones until the latter are consumed or expired. Hence resources used by such expired messages will not be freed, and they will be counted in queue statistics (e.g. the number of messages in the queue)."

Since the TTL is at a queue-level (all queues in my case), they should all expire, correct? There shouldn't be a message at the head of the queue that would hold up other 'expired' messages behind it.

In any case, I did try starting my consumer app, and all of those messages got consumed - they did not get dead lettered or expire. That indicates that the TTL did not apply.

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/JVCXAn3Q3hI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[Michael Klishin]

The docs say that messages are deleted when they reach the head of the queue, not earlier.

I tried reproducing this and strongly suspect that this particular behaviour is due to a retroactively

applied policy. I cannot reproduce anything distantly similar with queues declared once

I have a policy in place.

Per-message and queue TTL are more predictable because messages (and queues) have

all the "expiration context".

[alam86]

Hi Michael,

Thanks for trying to reproduce this! What you said made me try recreating the queue, and that fixed it. Interestingly, I didn't have to delete all my queues. Just deleting one queue (which got recreated since my consumer was still running) fixed the problem for all queues. All  messages on all queues are expiring as expected now!

A little unexpected - but you would think recreating the queue would not be necessary. Especially since the start of the TTL document says this: 

"When a TTL policy is being applied to a queue which already has messages in it, matching messages will be discarded immediately (with some caveats)."

This indicates that the policy should apply to existing queues too (that may have messages in them). 

Anyway, thank you for your help with this!

MK

Staff Software Engineer, Pivotal/RabbitMQ

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/JVCXAn3Q3hI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Queues are stateful entities. They handle policy changes that affect them but there isn't always

enough information to do that for TTL: in particular,

we do not know when a particular message was enqueued *unless you use the expiration property*,

so we don't know how to compute that value.

I will try to clarify this in the docs.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

We've updated the docs:

http://www.rabbitmq.com/ttl.html#per-message-ttl-caveats

Hopefully it is an improvement and will spare future beginners some confusion.

[Parvez Lalani]

Thank you Michael!

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[Benjamin Dodd]

Hi,

I am experimenting with this at the minute.

I have encountered the same problem.

Now I need to pick a 'sacrificial lamb' queue to delete so the policy gets run across all queues.

I don't really want to delete any queues as my queues are set as Durable, and when the consumer comes back online I want it to be there. But I don't want the queue to build up so much that it crashes the system (which is what is happening).

I understand that better software design should stop the queue building up so much, but its harder for us to customize these per-message-TTL's per client when putting them in place at code-time. I see this is why policies were made to begin with.

[Benjamin Dodd]

Just a note here, The only way I could get the policy to work was removing the queue and adding it again with the policy already in place. Adding the policy after any queues are created had no effect on the TTL's. Using CloudAMQP.com with  RabbitMQ 3.9.13, Erlang 24.1.7

My workaround is to use a max-length policy instead, which worked on already created queues.

[Michal Kuratczyk]

Hi,

If you can create a reproduction script/provide the steps, we are certainly interested.

As for the issue you are trying to solve - message TTL as a solution to messages accumulating in the queues seems off.

If the messages have a natural TTL - they lose their business meaning/value after some time, then sure. But if you want

to prevent queues getting long, you should have more consumers or have a max queue length with either drop-head

or reject-publish. Or perhaps these messages should be in different queues in the first place, if they have a different

lifecycle.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/b60a394d-af85-4058-8e09-8c6bd3f407d9n%40googlegroups.com.

--

Michał

RabbitMQ team