question about using multiple subscribe blocks.

[Matt Murphy]

Hello,

I'm using amqp as follows:

https://gist.github.com/eb93aa1856f421916a87

I was under the impression that the work being done in each subscribe
block will happen simultaneously. Is this true? Oddly, the behavior
of my app is that all of the messages go into the 'first' queue, and
then once all are done, the worker whose job it is to process messages
from first -> second queue starts getting messages, etc.

So in essence my workers are not doing anything in parallel even
though as far as I can tell they should be.

Would anyone be able to take a peek at the gist above?

I'd expect messages to appear in the 'second' queue right away and
then to be processed by worker 2 ...

Any advice would be much appreciated.

-Matt

[Aman Gupta]

The problem here is that rabbit will send you all the messages on the
first queue as soon as you subscribe, so you have a big backlog of
messages to consume on the connection.

The correct solution here is to use prefetch, but that is only
available in the upcoming 1.6 rabbitmq release (you can build it from
their hg repository if you want). With prefetch, you would do:

mq = MQ.new
mq.prefetch(1) # grab only one un-acked message at a time

mq.queue('first').subscribe(:ack=>true) do |headers, msg|
process(msg)
headers.ack
end

With rabbitmq 1.5, you can try subscribing on two different channels
(I'm not sure if this will help, but it might):

fmq = MQ.new
fmq.queue('first')....

smq = MQ.new
smq.queue('second')....

If multiple channels doesn't help and you're unable to upgrade
rabbitmq, you'll have to use pop instead of subscribe:

mq = MQ.new
f = mq.queue('first')
f.pop do |msg|
if msg
process(msg)
f.pop # get next message
else
# no message, poll again in one second
EM.add_timer(1){ f.pop }
end
end

Aman

[Matt Murphy]

Aman,

One quick question -- is it correct to assume that setting prefetch to some value greater than 1 but smaller than "all" would also work and might allow some messages to be prefetched while still obtaining my desired behavior?  

[Aman Gupta]

On Wednesday, June 17, 2009, Matt Murphy <mmm...@gmail.com> wrote:
> Aman,
> One quick question -- is it correct to assume that setting prefetch to some value greater than 1 but smaller than "all" would also work and might allow some messages to be prefetched while still obtaining my desired behavior?

Yep.