Dead-letter exchange and republishing to origin queue

[hed...@gmail.com]

(note: this was originally posted on rabbitmq-discuss, but I feel rabbitmq-users is a more appropriate place for this post ?)

Hello everyone,

I recently dabbled with dead letter exchanges in rabbitmq 3.5.6 and php-amqplib. My use case is : when a consumer rejects a message, it goes into a "retry queue", where it must wait X secs before being redelivered in the original queue.

I have the following rabbitmq config:

• Direct exchange named "e.main" + binding to "q.main" queue.
• Retry exchange named "e.retry" + binding to "q.retry" queue.
  
• "q.retry" defines x-message-ttl to TTL
  
• "q.main" defines x-deadletter-exchange to "e.retry"
  

With this config, it's widely accepted that a message rejected in "q.main" will never be redelivered to "q.main" after staying for TTL in 'q.retry" (the message is dropped). As defined in https://www.rabbitmq.com/dlx.html:

It is possible to form a cycle of dead-letter queues. For instance, this can happen when a queue dead-letters messages to the default exchange without specifiying a dead-letter routing key. Messages in such cycles (i.e. messages that reach the same queue twice) will be dropped if the entire cycle is due to message expiry.

So I'm changing things around and here's a config where the message can be redelivered in the origin queue

• Direct exchange named "e.main" + binding to "q.main" queue.
• Retry exchange named "e.retry" + binding to "q.retry" queue.
  
• "q.retry" defines x-message-ttl to TTL
  
• "q.retry" defines x-deadletter-exchange to "e.main"
• The consumer must manually ACK from q.main and publish to e.retry.

When TTL is elapsed, the message goes back to q.main.

Is it a good practice ? The immediate drawback is that you can define only one x-deadletter-exchange - meaning that multiple queues must define separate "retry exchanges". Also, the first config is more intuitive to understand, and each queue could define the same "retry exchange".

Also, the rule defined in rabbitmq docs specify "that the entire cycle must be due to message expiry". Is a basic_reject a form of message expiry?

What is the commonly accepted way to redeliver a message to its origin queue after a "retry ttl" ?

Thanks,

[Laing, Michael]

Did you consider this? https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paulo Almeida]

Hi,

This is how I have implemented retries (this is used to post events to a callback URL):

1. A queue named "event_send" created with the following attributes:

    x-dead-letter-exchange: "" (default exchange)

    x-dead-letter-routing-key: "event_send.retry"

    durable:true

2. A queue named "event_send.retry" created with the following attributes:

  x-dead-letter-exchange: "" (default exchange)

  x-dead-letter-routing-key: "event_send"

  x-message-ttl: 600000

  durable: true

3. Multiple topic exchanges (one per event type) bound to queue "event_send"

The consumer of "event_send" then only needs to ack if the event was successfully handled and reject if it needs to go to the retry queue. After 10 minutes, each message in the retry queue will move back to the main queue.

This works because I'm using topic exchanges when initially publishing the message (no routing key needed). Note that the routing key will be overwritten when it first expires and is routed via the default direct exchange. The solution also assumes infinite retries. There is no way of limiting the number of retries.

[HED]

Using the default exchange + dead-letter-routing-key is indeed a good
idea because the default exchange has access to any queue (whatever
the exchange types are).

I wonder why the "naive" implementation doesn't work "out of the box"
and why we have to design alternative solutions around default
exchanges, additional plugins (as Michael Laing suggested) etc.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "rabbitmq-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/rabbitmq-users/X6pYAP_v4ew/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to