RabbitMQ - Dead lettering on Queue Limit - Dead letter recent messages after queue size limit

[Vidhya rani]

Hi all,

I have configured Dead letter exchange for all my queues based on queue limit - https://www.rabbitmq.com/dlx.html

I have created a policy, with maximum queue limit and Dead letter exchange. These are the parameters - 

 

dead-letter-exchange:	Dead-Exchange
max-length-bytes:	1000

With the above policy, If there are any messages after 1000 bytes,the first message which was present in queue is going to Dead-Exchange and getting routed as per original key. 

I want the new message which has come after the queue has reached 1000 bytes, to go to dead letter queue. Based on this documentation https://www.rabbitmq.com/maxlength.html, I added one more parameter to the policy

dead-letter-exchange:	Dead-Letter-Exchange
max-length-bytes:	1000
overflow:	reject-publish

With this policy, once my queue size has reached 1000 bytes, the newer messages are rejected. It is not going to dead letter exchange. 

Can someone suggest how to dead letter newer messages instead of the first messages in queue. 

Thanks in advance

[Stuart Johnston]

Queue Overflow Behaviour

Use the overflow setting to configure queue overflow behaviour. If overflow is set to reject-publish, the most recently published messages will be discarded. In addition, if publisher confirms are enabled, the publisher will be informed of the reject via a basic.nack message. If a message is routed to multiple queues and rejected by at least one of them, the channel will inform the publisher via basic.nack. The message will still be published to all other queues which can enqueue it.

To define an overflow behaviour - whether to drop messages from head or to reject new publishes, add the key overflow to a policy definition. For example:

rabbitmqctl	rabbitmqctl set_policy my-pol "^two-messages$" '{"max-length":2,"overflow":"reject-publish"}' --apply-to queues
rabbitmqctl (Windows)	rabbitmqctl.bat set_policy my-pol "^two-messages$" "{""max-length"":2,""overflow"":""reject-publish""}" --apply-to queues

The my-pol policy ensures that the two-messages queue contains no more than 2 messages and all additional publishes are sent basic.nack responses as long as the queue contains 2 messages and publisher confirms are enabled.

Policies can also be defined using the management plugin, see the policy documentation for more details.

[Vidhya rani]

Hi Stuart

I have used overflow behaviour. As mentioned in my post, with overflow setting, new messages are getting discarded. But they are not going to Dead letter queue. I want the new messages to go to the Dead letter queue

Do we have this behaviour in newer version of RabbitMQ or is there any bug for this ? 

Requirement:

If I configure queue limit on my queues, and if my queue limit is reached, any newer messages that come after the queue limit should go to configured Dead letter queue. 

[Stuart Johnston]

I think you need to do the following by setting the requeue to false

Messages from a queue can be "dead-lettered"; that is, republished to an exchange when any of the following events occur:

• The message is negatively acknowledged by a consumer using basic.reject or basic.nackwith requeue parameter set to false.
• The message expires due to per-message TTL; or
• The message is dropped because its queue exceeded a length limit

The overflow only has two arguments drop-head or basic-reject so if you set overflow to basic reject and requeue to false 

On Tuesday, 4 June 2019 08:07:31 UTC+1, Vidhya rani wrote:

[Vidhya rani]

Hi Stuart

Thanks for the response. 

Sorry for asking late. I see requeue parameter present as part of basicReject / basicNack. In this case, the consumer would be default consumer by RabbitMQ when the queue size limit is reached. 

In policy i checked if i can add the requeue parameter, I did not find and option to do that. Can you please let me know where should i add the requeue property for dead letter overflow

Thanks 

[Michael Klishin]

The requeue parameter comes from your consumers. To put it differently, it's a property of a client-sent protocol frame. You cannot

set it via a policy. It is not a queue property or optional arg.

1. https://www.rabbitmq.com/confirms.html#consumer-nacks-requeue

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/bf722f61-d46f-4b65-aef2-5cb998153402%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Vidhya rani]

Hi Michael,

It was earlier mentioned in thread to set requeue as false  and overflow behaviour to reject publish. 

My requirement was once the queue limit is reached, any new messages should go to dead letter queue instead of older messages going to dead letter queue. 

Since consumer for this case(Dead lettering) would be internal/default, I am confused on where to set the requeue property

On Monday, July 1, 2019 at 3:26:01 AM UTC+5:30, Michael Klishin wrote:

The requeue parameter comes from your consumers. To put it differently, it's a property of a client-sent protocol frame. You cannot

set it via a policy. It is not a queue property or optional arg.

1. https://www.rabbitmq.com/confirms.html#consumer-nacks-requeue

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/bf722f61-d46f-4b65-aef2-5cb998153402%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

If overflow behavior is "reject publishes" it does not matter what your consumers do. Rejecting without

requeueing will dead letter messages regardless of the overflow mode (or even if queue length limit is set up at all).

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/26c20010-f1ac-49e1-8d3f-fd8e1cbc0ebc%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.