Dead-letter exchange and republishing to original queue

[prisca polyte]

Hello, 

We implement a delayed retry system based on rabbitmq

We have a pending queue  (named PENDING) that contain messages that have to be delivered .

On the other side we have an other queue we use as a dead letter queue named PENDING_DLQ. This queue is bound with a dlx header exchange.

PENDING_DLQ has a TTL to 2 min (x-message-ttl: 120000). PENDING queue is declared as dlq for PENDING_DLQ

The consumer consume messages from PENDING and ack them. If anything goes wrong, we put the messages in PENDING_DLQ. 

Normally, after TTL expires messages from PENDING_DLQ are sent back to PENDING.

This process works like a charm for 3 weeks.

Now, we see stuck messages in PENDING_DLQ. They never leave the PENDING_DLQ queue. everything is blocked.

Though the UI, if we consume and requeue 100 messages from the head of the queue (PENDING_DLQ) it frees the next ones.

Any idea to solve this ?

thx

Regards

[prisca polyte]

Hello,

We found the source of the issue. 

Unfortunately our DLQ (PENDING_DLQ ) was a priority queue. That was the mistake. 

As the dlq was a priority queue,  at the head there were the most priority messages. 

Once the ttl expires the messages went back to pending queue at the head of the queue. We tried to deliver them unsuccessfully . so they wen back to DLQ . 

So, we ended up looping over few messages that for some reason cannot be delivred.

After a while these messages exppired (business rule), all messages behind were kind of free but they were also expired.

Son we recreate the DLQ with no priority and everything go back to normal.

thx