Ensure chronological order of consumed messages

[Sam Van der Borght]

Hi,

Im trying to find out if rabbit is a good solution for the following problem: I'm trying to process transactions in a chronological order, i must guarantee that the oldest transaction from a user is processed. 

A transaction is linked to a specific user so i thought about declaring a named queue for each user that has transactions that need processing.

However i read that rabbit doesnt load balance in queues but in consumers, which mean my transactions will be in the correct order in the queue but they will be consumed at the same time and it might be that transaction 3 finished before trans 1.

it's like a blocking queue that waits for the consumer to be finished before it releases another message, is there something like that?

Can rabbit handle this problem? How would i go about solving this?

tl;dr

I want to make a private queue for each customer and handle all his transactions in chronological order. How can i do this?

[Michael Klishin]

On 6 August 2014 at 16:30:16, Sam Van der Borght (samvand...@gmail.com) wrote:
> > Im trying to find out if rabbit is a good solution for the following
> problem: I'm trying to process transactions in a chronological
> order, i must guarantee that the oldest transaction from a user
> is processed.
> A transaction is linked to a specific user so i thought about declaring
> a named queue for each user that has transactions that need processing.
> However i read that rabbit doesnt load balance in queues but in
> consumers, which mean my transactions will be in the correct
> order in the queue but they will be consumed at the same time and
> it might be that transaction 3 finished before trans 1.

Only if you have more than 1 consumer. Consumer can be added as exclusive.

Note that in the presence of failing consumers, messages will be re-queued
and while RabbitMQ tries to preserve the original order, in some cases it may
not be sufficient.

> it's like a blocking queue that waits for the consumer to be finished
> before it releases another message, is there something like
> that?
>
> Can rabbit handle this problem? How would i go about solving this?

RabbitMQ queues are FIFO. See tutorial 2:

http://www.rabbitmq.com/getstarted.html

setting basic.qos prefetch = 1 with an exclusive consumer
will give you the "1 at a time" behaviour you want.

What you should worry about is consumer failures and issues
with consumer unavailability (e.g. a consumer getting stuck or into a livelock/infinite loop) when there's only 1 consumer allowed.

These problems are not really RabbitMQ-specific and consumer acknowledgements
+ prefetch = 1 give you a good start on a solution. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Sam Van der Borght]

Thanks Michael, that was a quick and informative answer!

I'm gonna try to make a setup like you described to see if anything else comes up!

Cheers