Is it possible to order messages by timestamp in priority queue?

1,139 views

Skip to first unread message

Jerry Lin

unread,

Dec 23, 2019, 12:07:39 PM12/23/19

to rabbitmq-users

I see the Priority Queue Support doc already describes the x-max-priority option well.

However, it seems not suitable for that case that the messages are ordered by timestamps.

Here is my scenario that needs this kind of feature.

Suppose you have a real time image processing service, starting from many camera stream producers (P), then processed by processors (W) and finally sink to consumers (C).

Processors (W) are stateless. It can benefit from data parallelism by launching multiple W sharing one worker queue.

Any one consumer (C) fetches a specific camera stream passing thru W, requiring that the input stream should be ordered by timestamp.

However, if the processor W latency varies, it does not guarantee the output stream is timestamp-ordered,

because it's possible that partial of specific stream is processed by distinct processors W.

It let me wonder if RabbitMQ can establish priority queues among W and C, that sort the messages by timestamps.

The x-max-priority option would not be relevant here since it has 255 limit.

Michael Klishin

unread,

Dec 24, 2019, 12:28:36 AM12/24/19

to rabbitmq-users

An intentionally limited number of priorities and traditional FIFO queues are the only options. Set the timestamp field of every message

and make your consumers reorder them after consuming a batch of N, or use a data store that you can query using any condition necessary

and limit RabbitMQ to passing around event notifications.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/47f051fa-8773-40e6-952e-b957ebd63923%40googlegroups.com.