Dequeue order when a consumer is subscribed to multiple queues

[Steve]

Hi

I'm having issues trying to get RabbitMQ to behave the way I want when a consumer is bound to multiple shared queues. First of all to give some context I'll explain my requirements. There are various different types of tasks/jobs that are processed by multiple workers/consumers. A worker can only process a single task at any given time and may only support a subset (one or more) of the different types of outstanding tasks. Ideally each task should be processed in the order they were sent regardless of the type. However a task can be processed earlier than expected by an idle worker if it doesn't support any of the tasks at the front of the queue (i.e. each worker should process the first task available of the type(s) it supports). To solve this with RabbitMQ I've used the following configuration:

This almost fulfils the requirements, the only issue is with regards to the ordering which does not occur in sequence if the consumer supports more than one type of task. This is especially apparent if a single consumer supporting all task types is running. In this scenario RabbitMQ appears to round-robin between each queue it is bound to, although it may take more than 1 item from a given queue before switching over to another one. For example if the following sequence of messages are published where A & B are the types:

1: A

2: B

3: B

4: B

5: A

6: A

The consumer could dequeue them in the following order:

1: A

5: A

2: B

6: A

3: B

4: B

The order is consistent for each queue when considered individually, however when combined it varies.

1. How does RabbitMQ decide the order in this scenario and can it be changed?

2. Is there a different configuration I haven't considered that will operate as required?

Any help/advice would be greatly appreciated! 

Thanks

[Michael Klishin]

On 24 October 2014 at 18:57:48, Steve (steph...@electrum.co.uk) wrote:
> This is especially apparent if a single consumer supporting
> all task types is running. In this scenario RabbitMQ appears
> to round-robin between each queue it is bound to, although it
> may take more than 1 item from a given queue before switching over
> to another one.

RabbitMQ round robins deliveries between consumers on a queue. However,
consumers can be added on separate channels and those channels can have different
basic.qos prefetch values, and consumers can have priorities [1].

Finally, there are [2] and [3].

If a consumer consumes from multiple queues, RabbitMQ does not try to interleave deliveries from different queues.

1. http://www.rabbitmq.com/consumer-priority.html
2. http://www.rabbitmq.com/consumer-prefetch.html
3. http://github.com/rabbitmq/rabbitmq-priority-queue
 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Steve]

Thanks for your reply.

With my configuration each consumer is on a separate server and uses a single channel for all of the queues it binds to. How does the consumer decide which of its bound queues to get the next message from if no priority has been set and it doesn't attempt to interleave?

The code I'm using is similar to this:

string exchangeName = "exchange-tasktype";
 
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
    using (var channel = connection.CreateModel())
    {
        channel.BasicQos(0, 1, true);
        channel.ExchangeDeclare(exchangeName, "direct");
        var consumer = new QueueingBasicConsumer(channel);
 
        string[] supportedTasks = new string[] { "A", "B" };
        foreach (string taskType in supportedTasks)
        {
            var queueName = "type-" + taskType;
            channel.QueueDeclare(queueName, false, false, false, null);
            channel.QueueBind(queueName, exchangeName, taskType);
            channel.BasicConsume(queueName, false, consumer);
        }
 
        while (true)
        {
            var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
            Console.WriteLine("Received {0}", ea.RoutingKey);
            channel.BasicAck(ea.DeliveryTag, false);
        }
    }
}

Thanks

[Steve]

Sorry Michael I've just read the Consumer Priorities link again and realised I missed the last example which is similar to my configuration (except a separate consumer per queue is used). If the channel and consumer limits are both set to 1 in that example, how does RabbitMQ decide which order to fetch the messages in?

[Michael Klishin]

RabbitMQ queues are FIFO with the exception of priority queues. With multiple consumers on a queue, the strategy is round robin.

MK

[Simon MacMullen]

On 27/10/14 09:03, Michael Klishin wrote:
> With multiple consumers on a queue, the strategy is round robin.

...but with multiple queues on a channel (with global prefetch limit),
it's random.

Cheers, Simon

[Steve]

Given what you've both said I think the closest I'll get to the requirements is by going with "the multiple consumers sharing the limit" example in combination with the priority. If the workers priority is based on the number of task types supported (i.e. workers with less support have higher priority) this should hopefully help spread the workload more evenly. The only issue is the ordering which we can hopefully live with.

Thank you both for your help

[Hongyu Zhang]

Sorry to pick up an old thread. I think it's a topic that also puzzled me recently, see my question posted on github, https://github.com/videlalvaro/php-amqplib/issues/293. Basically I didn't see issue with the Java client, but the PHP client gave me random output. 

So if a client does need to read/write to multiple queues, what's the best practice here? Do we need to create multiple channels, each queue in one channel?

Thanks.

[Michael Klishin]

On 4 November 2015 at 23:40:32, Hongyu Zhang (ma...@hongyu.org) wrote:
> So if a client does need to read/write to multiple queues, what's
> the best practice here? Do we need to create multiple channels,
> each queue in one channel?

As far as consumers go, Java, .NET and perhaps more clients guarantee ordering
per channel. I'm not sure how it can work across channels, which can have different
QoS prefetch values.

I'm not familiar with the PHP one to comment. Multiple queues assume concurrency
and interleaved messages in general, regardless of the client, so how total ordering
is defined between N queues probably varies from person to person.

[Hongyu Zhang]

Actually, I am not trying to request the message ordering across channels. My sample code illustrated a fatal error in the PHP library, i.e., a messages written into the first queue was mistakenly retrieved out of the second queue.

[Michael Klishin]

On 5 November 2015 at 10:04:30, Hongyu Zhang (ma...@hongyu.org) wrote:
> My sample code illustrated a fatal error in the PHP library,
> i.e., a messages written into the first queue was mistakenly
> retrieved out of the second queue.

Messages can be routed to more than one queue. Please post a script that can demonstrate
what you're seeing. And start a new thread, while we are at it.