Re: [rabbitmq-discuss] Sequential message processing guarantee

[Alexandru Scvorţov]

Hi Yogesh,

In the absence of consumer failures, RabbitMQ will deliver messages from
a queue in order.

So, if messages 1, 2, 3 reach a queue in order, RabbitMQ will deliver
them to consumers in the same order (1, 2, 3).

But, if consumers fail before acknowledging the messages (or if they
reject the messages), those messages will requeued at the end of the
queue. So, if the consumer that got message 1 rejects it, the new
order of messages will be 2, 3, 1.

This all deals with message *delivery*.

> There are multiple consumer threads consuming the messages.
> Though multiple threads are consuming the messages, I observed that
> messages still get processed sequentially.
> Is that the case and if NOT, what is the way to guarantee sequential
> processing of messages on one queue?

The library makes no guarantees about the order in which you process
messages, only about the order in which they're delivered.

If you want to process all the messages on a queue in order, only
consume from one thread (doing basic.get from one thread like you
suggested in the other email would work, but would also be highly
inefficient).

Does this answer your question?

Cheers,
Alex

On Sun, Jan 22, 2012 at 09:38:40AM -0800, Yogesh Ketkar wrote:
> Running the code below, gives
> channel.basicConsume(QUEUE_NAME, autoAck, CONSUMER_TAG,
> new DefaultConsumer(channel) {
> @Override
> public void handleDelivery(String consumerTag, Envelope
> envelope, BasicProperties properties, byte[] body) {
> System.out.println(Thread.currentThread().getName());
> }
> }
>
> o/p like
> pool-1-thread-1
> pool-1-thread-2
> pool-1-thread-3
> etc
>
> There are multiple consumer threads consuming the messages.
> Though multiple threads are consuming the messages, I observed that
> messages still get processed sequentially.
> Is that the case and if NOT, what is the way to guarantee sequential
> processing of messages on one queue?
>
> regards, Yogesh
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq...@lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Emile Joubert]

Hi Yogesh,

On 22/01/12 17:38, Yogesh Ketkar wrote:
> Is that the case and if NOT, what is the way to guarantee sequential
> processing of messages on one queue?

If I understand your question correctly, you want to know whether
messages will always be delivered in order. This page has relevant info:
http://www.rabbitmq.com/semantics.html#ordering :

"""
messages published in one channel, passing through one exchange and one
queue and one outgoing channel will be received in the same order that
they were sent
"""

This remains true whether you use basic.get or basic.consume. A consumer
may observe messages out of order if there are multiple consumers
consuming from the same queue and some of them perform requeues.

-Emile

[Yogesh Ketkar]

Thanks Alex.

I think, for sequential message processing, rather than this pattern

while(true) {

    GetResponse res = channel.basicGet(QUEUE_NAME, false);

    if(res != null) {

        // process and ack message

   }

}

using channel.basicConsume on the channel which is created like this

ExecutorService es = Executors.newSingleThreadExecutor();

Connection connection = factory.newConnection(es);

final Channel channel = connection.createChannel();

is probably a better approach.

Any comments on this?

regards, Yogesh

2012/1/23 Alexandru Scvorţov <alex...@rabbitmq.com>

[Alexandru Scvorţov]

Hi Yogesh,

> using channel.basicConsume on the channel which is created like this
> ExecutorService es = Executors.newSingleThreadExecutor();
> Connection connection = factory.newConnection(es);
> final Channel channel = connection.createChannel();
>
> is probably a better approach.
> Any comments on this?

I don't quite see the point of the ExecutorService.

Just doing this:

> Connection connection = factory.newConnection(es);
> final Channel channel = connection.createChannel();

Then, calling basicConsume with your own subclass of DefaultConsumer
should be fine.

Cheers,
Alex

[Yogesh Ketkar]

Hello Alex,

I thought, using default Connection con = factory.newConnection(),
ThreadPool which gets created will have more than one thread and
calling basicConsume will result in processing of messages on multiple
threads.

But if we explicitly provide Executors.newSingleThreadExecutor() to
newConnection method, only single thread will come into play with
basicConsume.

Is my understanding correct?

regards, Yogesh

> > 2012/1/23 Alexandru Scvorţov <alexan...@rabbitmq.com>

> > > > rabbitmq-disc...@lists.rabbitmq.com

> > > >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> _______________________________________________
> rabbitmq-discuss mailing list

> rabbitmq-disc...@lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Alexandru Scvorţov]

> Is my understanding correct?

Almost. Deliveries will happen serially on a channel, so, as long as you
only have one consumer, everything should be fine.

[Steve Powell]

Hi Yogesh,

What Alex says is perfectly correct.

The executor service is set on the connection, and is there to allow multiple

channels to execute consumers simultaneously, and without blocking the channel

thread. By default you get an executor service which has five threads.

From the point of view of a single channel, the consumers are driven serially,

one delivery at a time, upon one of the executor service threads. The same

thread is not guaranteed to be used each time for each consumer call for a

channel (so don't try to use ThreadLocal objects).

The reason we provide this, is so that clients with many, many channels are not

forced to have a Consumer thread per channel -- something that is quite

expensive -- but have the option of allocating as many threads as they wish if

they need it. At the same time we guarantee that no Consumer calls can

'overtake' each other on any one channel, and that Consumer processing runs on a

distinct thread from the channel and the app, so that channel calls in the

Consumer code will not deadlock.

I hope this clarifies rather than obscures the situation,

Regards,

Steve Powell  (a hoppy bunny)

----------some more definitions from the SPD----------

vermin (v.) Treating the dachshund for roundworm.

chinchilla (n.) Cooling device for the lower jaw.

socialcast (n.) Someone to whom everyone is speaking but nobody likes.

[Simone Busoli]

Alex, I'm quite confident that since 2.7.0 messages are kept in queues in the same order in which they reached the queue in the first place.

[Yogesh Ketkar]

Hello Steve/Alex,

Thanks a lot for the help. It's pretty clear now.

So basically for a queue, as long as there is going to be only one consumer, it hardly matters if ThreadPool with com.rabbitmq.client.Connection underneath the channel has 1 thread or many threads.

Well, having more threads will help if same channel is going to be used to consume messages from other queues.

regards, Yogesh

2012/1/23 Steve Powell <st...@rabbitmq.com>

[Yogesh Ketkar]

> Well, having more threads will help if same channel is going to be used to
> consume messages from other queues.

Actually more appropriate would be, having more threads will help if
same Connection is going to be used
with different channels.

> > Steve Powell  (*a hoppy bunny*)
> > *----------some more definitions from the SPD----------*
> > *vermin* (*v.*) Treating the dachshund for roundworm.
> > *chinchilla* (*n.*) Cooling device for the lower jaw.
> > *socialcast* (*n.*) Someone to whom everyone is speaking but nobody likes.

[Steve Powell]

Not only more appropriate, but also more true :-)

Steve Powell  (a happy bunny)

----------some more definitions from the SPD----------

vermin (v.) Treating the dachshund for roundworm.

chinchilla (n.) Cooling device for the lower jaw.

socialcast (n.) Someone to whom everyone is speaking but nobody likes.