Multiple consumers on one queue

[cbare]

Hi,

Please help a clueless noob!

I set up the example producer and consumer from the carrot
introduction docs here:

http://ask.github.com/carrot/introduction.html

That worked nicely, so I got brave and modified the producer to send
100 messages in a loop. The I tried two consumers reading off the same
queue. They seemed to trade off, with one consumer getting the even
messages and the other getting the odds.

Then I stuck a sleep(6) in one of the consumers to simulate a heavily
loaded worker. I still see the same even-odd behavior, which wasn't at
all what I was expecting. I thought it would give messages to the next
available consumer, maybe choosing randomly if more than one consumer
was available.

So, my question is: What am I doing wrong?

Thanks for any hints or pointers to the appropriate docs or source.

-chris

[Christopher Bare]

Thanks a ton, AJ. Setting consumer.qos(prefetch_count=1), I see the
behavior I expected with the slower consumer getting fewer jobs.
Perfect.

The docs warn against using consumer.wait() in production. Later in
the change log, it says that consumer.iterconsume() will be
deprecated. What's the proper production-ready way for a background
process to sit in a loop waiting for messages until further notice,
cnt-c or shutdown signal. If I had a quick example of that, I'd be
totally in business.

Thanks!!

-chris

On Sat, Jul 10, 2010 at 6:45 AM, filtered <zopyx...@googlemail.com> wrote:
> I think the behavior is related to the 'prefetch_count' which must be set to
> 1 in your case.
> There is some code in carrot (qos() method where you can set the prefetch
> count but I don't know how how and where to call this exactly.
> -aj
>
> 2010/7/10 cbare <christo...@gmail.com>

>> --
>> You received this message because you are subscribed to the Google Groups
>> "carrot-users" group.
>> To post to this group, send email to carrot...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> carrot-users...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/carrot-users?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "carrot-users" group.
> To post to this group, send email to carrot...@googlegroups.com.
> To unsubscribe from this group, send email to
> carrot-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/carrot-users?hl=en.
>

[Ask Solem]

On Jul 10, 2010, at 6:56 PM, Christopher Bare wrote:

> Thanks a ton, AJ. Setting consumer.qos(prefetch_count=1), I see the
> behavior I expected with the slower consumer getting fewer jobs.
> Perfect.
>
> The docs warn against using consumer.wait() in production. Later in
> the change log, it says that consumer.iterconsume() will be
> deprecated. What's the proper production-ready way for a background
> process to sit in a loop waiting for messages until further notice,
> cnt-c or shutdown signal. If I had a quick example of that, I'd be
> totally in business.
>

The most flexible way is to use connection.drain_events:

consumer.register_callback(callback)
consumer.consume()

while True:
connection.drain_events()

drain_events also supports a timeout argument, which is very useful
in some cases, and it will consume from all active channels, not just the
one used by the consumer.

I'm currently rewriting carrot with a new API. It's called kombu:
http://github.com/ask/kombu
and drain_events will be the only way to consume there.

--
{Ask Solem,
Developer ~ Webteam,
Opera Software,
+47 98435213 | twitter.com/asksol }.