Quorum queues: Possible to "peek" at messages without changing the order?

[David Ernstsson]

Use case: We would like to "peek" at the messages in our dead letter queue without changing the order. How can we accomplish this?

We do not have access to rabbitmq management portal due to performance issues (nor should it by documentation be used in production)  

From documentation https://www.rabbitmq.com/confirms.html

"When a message is requeued, it will be placed to its original position in its queue"

When it comes to quorum queues this does not seem to be accurate at all and instead message is always placed at the tail (with header x-delivery-count increased). 

Questions:

1. Is the quorum queue redelivery at the end of the queue rather than the front intended or a mere internal side effect?

2. Is it at all possible to "peek" at some messages in the queue and then nack+redeliver them without changing the order?

Best regards, David

[Karl Nilsson]

Hi,

There is no way to non-destructively peek at messages in the queue. For quorum queues at least it would be relatively easy to implement as a rabbitmqctl command but just not something we've got around to yet.

It is true that quorum queues take a less strict approach to returned messages. Internally it keeps a single separate queue for returned messages which is emptied before the main queue. So once a message has been returned it will be served before other messages. That said the return queue works as a simple FIFO queue so once it has been returned once it will join the back of the returns queue, as you have noticed.

Ordering is a tricky concept with RabbitMQ queues and the only real invariant we can provide is that messages from a single producer will reach the queue in the order they were sent. Given queues allow for competing consumers and returns the ordering after that is pretty much non-deterministic. At the very least it should never be relied upon when writing applications that use RabbitMQ.

When ordering is important only really the single active consumer feature provides anything like ordering when it comes to consumption.

I appreciate this isn't what you were asking but perhaps it provides a bit of justification for the implementation of quorum queues. It is simpler and more efficient to give up on ordering at the point where no guarantees can be provided anyway.

I will consider adding a peek command to the rabbitmq-queues tool that works on quorum queues but can't promise when or even if that may happen.

Cheers

Karl

---

From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of David Ernstsson <david.e...@gmail.com>
Sent: 02 September 2020 5:40 PM
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: [rabbitmq-users] Quorum queues: Possible to "peek" at messages without changing the order?

 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/89df8e66-aa0b-460f-9036-d076323f991fn%40googlegroups.com.

[David Ernstsson]

Hi Karl and thanks for your detailed answer,

Being able to keep the order of messages here is mostly just from debugging/error/retrying logic when it comes to either have a look at a faulty queue or our dead-letter queue where we would (at least as little as possible) being able to peek once, twice, multiple times without destroying the order completely. That ordering is tricky and nothing to rely on is of course very true.

That being said your reasoning is sound and if we want to peek and do more advanced stuff with our dead-letter queue we can always build some persistent storage ourselves around it and deal with it in a non-rabbitmq native way from the start (as it should be arguably)

That also being said we would certainly like to have more features/commands available in the rabbitmqctl where this could be one very nice feature. Rabbitmqctl will already be a vital part of our maintenance/debugging stack

Cheers, David

[Karl Nilsson]

For a "peek" command would you envisage this just showing the next message in the queue or more like a "tail" command where you can show the next n messages in the queue?

---

From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of David Ernstsson <david.e...@gmail.com>

Sent: 03 September 2020 8:51 AM
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: Re: [rabbitmq-users] Quorum queues: Possible to "peek" at messages without changing the order?

 

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/3edacb02-21da-4034-8c34-ba5e08735eadn%40googlegroups.com.

[Simone B]

I second this request.
A way to peek into a quorum queue would be very useful; as of now I find it very hard to skip items in a quorum queue. The only way that I've found to do it is extremely slow.

Reason #1: I've already been trying to make a "peek" procedure to list the queue contents (for the user reading leisure). I had to get messages one by one using the RabbitMQ interface instead of the AMQP interface, otherwise I couldn't get around infinite loops. The docs discourage this for performance reasons, but I haven't found another way. It also needs to reject each message, so it's not really 100% "peeking" as it might mix the messages up a bit.

Reason #2: using the AMQP interface for supposedly better performances is instead causing me to go to great workarounds.  In one of my use cases, messages must be processed after a certain delay; think for example of a trashcan where you want to remove stuff after it spends a couple of days there. As using "reject" is causing infinite loops, instead I've been trying to republish-and-ack messages, but this is reeeeaaally slow (like, single digit messages per second).

Reason #3: If there was any scenario where there is a guarantee that the message ordering in the queue does never change, then there would be several possible performance improvements on the application side. In the previous "trashcan" example there are no strict ordering requirements; the trashcan can be emptied in any order, as long as the objects spend a certain amount of time there. A weaker requirement would be that, for every consumer, when the consumer skips from a message to the next message, the next message is published after the first message (i.e. no global order, only per-thread order). In this way, when a message is received, the consumer could just sleep until the message is "old enough". This would avoid scanning the whole queue uselessly looking for messages. But as far as I know this is not doable because even if the consumer does never reject or republish a message, there is no guarantee that if the consumer crashes the message is not moved in a different position in the queue (I would like to be proven wrong).

So, there ... it would be 3 improvements at once. Unless I'm missing another way to accomplish this...

Best regards,
Simone

[Diana Corbacho]

Hi all,

There will be a peek command in RMQ 3.8.10, see [1]

`rabbitmq-queues peek [--vhost <vhost>] <queue> <position>`

[1] https://github.com/rabbitmq/rabbitmq-cli/pull/452

Kind regards,