Help with implementing Priority Queue in Pika

[Pythonjunkie]

I have 3 queues q1, q2 and q3 and 3 consumers c1, c2 and c3.

q1 has high priority messages and q2 has the next priority.

c1 listens to only q1 and if c1 is taking time to finish the tasks, c2 should be able to take up some tasks from q1.

I have currently bound c2 to both q1 and q2 and used 'x-priority' to set up a higher priority for q1 and a lower priority for q2 using:

"channel.basic_consume( msg_consumer, queue="q1", arguments={'x-priority':10})"

"channel.basic_consume( msg_consumer, queue="q2", arguments={'x-priority':5})"

If I run c2 alone, it computes the tasks from q1 first and after all the tasks from q1  are completed, computes the tasks from q2, and works as expected.

But if I run both c1 and c2 in parallel, c2 just skips the taks from q1 and computes the tasks from q2, maybe because c1 is already listening to q1. This is not what I intend to do.

When both c1 and c2 are running in parallel, I would like both of them to compute tasks from q1 first.

Can anyone suggest me a way that I could parallelise this process?

Thanks

[Michael Klishin]

Sounds like http://github.com/rabbitmq/rabbitmq-priority-queue may do what you
want. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Simon MacMullen]

On 22/10/14 19:01, Pythonjunkie wrote:
> I have currently bound c2 to both q1 and q2 and used 'x-priority' to set
> up a higher priority for q1 and a lower priority for q2 using:
> "channel.basic_consume( msg_consumer, queue="q1",
> arguments={'x-priority':10})"
> "channel.basic_consume( msg_consumer, queue="q2",
> arguments={'x-priority':5})"

Note that consumer priorities are scoped within a queue - they allow
queues to determine which consumers to favour, not consumers to
determine which queues to favour.

Michael is correct in that the priority queue plugin is probably what
you want.

Cheers, Simon

[Anurag]

Is there a way to do it in python, using pika?

[Michael Klishin]

 On 24 October 2014 at 02:25:35, Anurag (anuragpa...@gmail.com) wrote:
> Is there a way to do it in python, using pika?

You can use the priority queue plugin with any client.

[Anurag]

I am not getting what I want with the priority key plugin.

I would like to know if 2 consumers in RabbitMQ can process different messages from the same queue in parallel?

i.e. if c1 starts processing messages from q1, and I run c2, I am unable to make c2 processes messages from q1.

Is there a way to do this?

Thanks

On Wednesday, October 22, 2014 2:01:52 PM UTC-4, Anurag wrote:

[Gavin M. Roy]

On Friday, Oct 24, 2014 at 2:44 PM, Anurag <anuragpa...@gmail.com>, wrote:

I am not getting what I want with the priority key plugin.

I would like to know if 2 consumers in RabbitMQ can process different messages from the same queue in parallel?

i.e. if c1 starts processing messages from q1, and I run c2, I am unable to make c2 processes messages from q1.

Is there a way to do this?

Yes, run two consumers against the same queue.  

[Anurag Patibandla]

But when I run c1 first and then c2, c1 starts consuming messages, and c2 just stops.

If I just run c2, it consumes messages from the queue. This is why I am confused

[Anurag]

I am using 'channel.basic_consume' to consume messages

[Gavin M. Roy]

On Friday, Oct 24, 2014 at 2:58 PM, Anurag <anuragpa...@gmail.com>, wrote:

I am using 'channel.basic_consume' to consume messages

On Friday, October 24, 2014 2:50:31 PM UTC-4, Anurag wrote:

But when I run c1 first and then c2, c1 starts consuming messages, and c2 just stops.

If I just run c2, it consumes messages from the queue. This is why I am confused

And you’re running this in two separate processes?

[Anurag Patibandla]

Yes. If I just run c2, it starts consuming messages from the queue. And if I just run c1, it starts consuming messages fromthe queue. But if I run both c1 and c2 in two separate processes, whichever I run first starts consuming messages from the queue and the other one is idle.

[Gavin M. Roy]

On Friday, Oct 24, 2014 at 3:05 PM, Anurag Patibandla <anuragpa...@gmail.com>, wrote:

Yes. If I just run c2, it starts consuming messages from the queue. And if I just run c1, it starts consuming messages fromthe queue. But if I run both c1 and c2 in two separate processes, whichever I run first starts consuming messages from the queue and the other one is idle.

This is not a normal behavior. Are you using Basic.QoS or exclusive queues? If you can share an example of your code with me on http://gist.github.com I’ll take a look and see if I can identify your problem.