Priority Queues, Per Message TTL and Dead Letter Queues

[deadbo...@gmail.com]

We are trying to use a priority queue to exponentially back off background processing jobs. We are letting high priority / low TTL messages dead letter back to the work queue. The odd behavior we see is if a low priority, long ttl message is sitting in the queue, if a new, high priority/low ttl message gets queued, it will not dead letter until after the low priority/high ttl; however if we ask for the next queue item, we get the high priority item, and then all messages behind it expire in priority order as expected. 

Is this expected behavior? How can queueing a new message replace the head item so it dead letters appropriately?

[Michael Klishin]

TTL timers aren't aware of priorities. Messages are dead lettered only when they reach queue head.

I'm not sure if designing a system around TTL and priority interplay is a good idea.

On 1 jul 2016, at 23:43, deadbo...@gmail.com wrote:

We are trying to use a priority queue to exponentially back off background processing jobs. We are letting high priority / low TTL messages dead letter back to the work queue. The odd behavior we see is if a low priority, long ttl message is sitting in the queue, if a new, high priority/low ttl message gets queued, it will not dead letter until after the low priority/high ttl; however if we ask for the next queue item, we get the high priority item, and then all messages behind it expire in priority order as expected. 

Is this expected behavior? How can queueing a new message replace the head item so it dead letters appropriately?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[deadbo...@gmail.com]

How do you reach the queue head if your priority is higher than the one currently at the queue head? Do message get prioritized before reaching the queue head, but ignore priority once they are there?

[Michael Klishin]

There is a separate backing queue per priority. They actually operate messages. So messages with the same priority are

"grouped" together if you will, and the queue process operates on the "bins" in order, so as long as there are high priority messages

to deliver and such, other priorities sit there waiting.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[deadbo...@gmail.com]

Thanks for your replies. I understand what's going on, but I don't see how TTL works with priority queues Let me give an example. If we have a queue with max priority 2 that dead letters into another queue. When I enqueue a message with priority 1, with a 3 minute TTL, followed by several message with priority 2 with a TTL of 10 seconds, nothing dead letters into the expected queue until after 3 minutes; however, if I ask for the next message (before the 3 minutes), I get the high priority/low TTL message and all messages dead letter as expected. 

Following your explanation, do all "bins" get a TTL timer on their respective head, or does only one message in the whole queue, get a timer? Is it possible to reprioritize that timer after messages have been enqueued?

Assuming this is expected behavior, what is the recommended implementation of backing off message retries?

[Chris Toomey]

Hi, just found this thread as we're trying to do the same thing as the OP (use a priority queue with TTLed messages for exponential backoff). We discovered the same surprising thing, that higher-priority, shorter-TTL messages aren't getting expired and dead-lettered until after lower-priority, longer-TTL messages that were published to the queue before them.

So in terms of how TTLs are enforced on priority queues, it looks like the implementation is the same as for unprioritized queues, namely in FIFO order irrespective of priority, even though the head of a PQ is by definition the highest-priority item in it. I.e., this behavior doesn't adhere to the described TTL behavior of expiring messages when they reach the head of the queue.

Exponential backoff is a pretty common thing to need to be able to do in distributed systems. Will the RMQ team consider changing how TTL works on priority queues to support those of us wanting to do that?

thx,

Chris

[Michael Klishin]

We will but our team of 8 has about 20 fairly high priority items and it is not currently in the top 5.

What has to change is not so much how TTL works but rather how priority queues are implemented.

[Chris Toomey]

Thanks Michael. I've filed https://github.com/rabbitmq/rabbitmq-server/issues/1449 for this.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/M72fH0hMB4Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.