[rabbitmq-discuss] Reordering messages
Dimitri del Marmol
I am facing the following problem: I have to process messages from a
queue but messages can be of either high or low priority.
Messages have to be processed FIFO except that all high priority
messages must be processed before the oldest low priority message is
processed.
The 'dumb' solution would be to reorder the list before processing
each message. I could do that by acking all the messages and reposting
them to the queue in the right order I guess...
Is there an established pattern for this?
Dimitri
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
David Wragg
> I am facing the following problem: I have to process messages from a
> queue but messages can be of either high or low priority.
> Messages have to be processed FIFO except that all high priority
> messages must be processed before the oldest low priority message is
> processed.
>
> The 'dumb' solution would be to reorder the list before processing
> each message. I could do that by acking all the messages and reposting
> them to the queue in the right order I guess...
>
> Is there an established pattern for this?
Put the two categories of messages onto two separate queues, and write
your application to preferentially consume from the higher priority
queue.
David
--
David Wragg
Staff Engineer, RabbitMQ
SpringSource, a division of VMware
Dimitri del Marmol
> your application to preferentially consume from the higher priority
> queue.
I actually toyed with the idea but I have trouble imagining what the
listener(s) would look like: should one listener subscribe to both
queues? If so, it will have to know when to ignore the low priority
callback (because the high priority queue is not empty)
David Wragg
Dimitri del Marmol <tonton...@gmail.com> writes:
>> Put the two categories of messages onto two separate queues, and write
>> your application to preferentially consume from the higher priority
>> queue.
>
> I actually toyed with the idea but I have trouble imagining what the
> listener(s) would look like: should one listener subscribe to both
> queues? If so, it will have to know when to ignore the low priority
> callback (because the high priority queue is not empty)
The details of how you achieve this depend on which client library you
are using, and how your application is organized. So with more
information someone might be able to give more specific advice. But
here is the general idea.
As you suggest, your listener should consume from both queues. You
should consume with no-ack=False (i.e. explicit acks).
In your code, you'll need a data structure to hold pending high-priority
messages (those that have arrived, but not yet been processed), and
another to hold pending low-prority messages. The basic logic is then
as follows:
When a message arrives (from either queue):
If a message is already being processed:
Add the new message to the approprite pending messages data structure
Else:
Start processing the message
When processing of a message completes:
Ack the processed message
If there are any pending high-priority messages
Remove one and start processing it
Else, if there are any pending low-priority messages
Remove one and start processing it
Else:
Nothing to do, so wait until another message arrives
So we obey the priorities because we always look for the high-priority
messages before looking in the low-priority slot. If there are always
high-priority messages coming in, then we never look at the low-priority
messages, though we may have many of them sitting there un-acked.
How you would code this up depends upon the client library, and whether you
are in a multi-threaded or event-driven environment.
Things are a bit more complicated if you want to have multiple workers,
all obeying the priorities.
David
--
David Wragg
Staff Engineer, RabbitMQ
SpringSource, a division of VMware