[bunny-amqp-devel] subscribing to multiple queues

[vishnu]

Hi Chris

    in our system we have a bunch of different messages that flow through the system. We were considering using several different queues for each different kind of message (so we can monitor the system easier), but signing up all consumers to subscribe to every queue so that we could automatically make sure we handle all messages efficiently. Is this possible with bunny?

thanks

Vishnu

[celldee]

Hi Vishnu,

I think that for one consumer to subscribe to many queues using bunny
you would have to employ some kind of multi-threaded processing model.
Each thread could use its own channel and reuse a single connection,
but I don't know what this would actually look like in code (assuming
that it could be written). Like I've said before, I have played a
little with bunny and threads, but I much prefer the idea of a number
of simple consumers rather than one complex one.

Regards,

Chris
_______________________________________________
bunny-amqp-devel mailing list
bunny-am...@rubyforge.org
http://rubyforge.org/mailman/listinfo/bunny-amqp-devel

[vishnu]

Hi Chris

   oh  I didnt mean in a multi-threaded model. It looks like the java client allows you to subscribe to multiple queues in a single thread. It seems to work because a subscription is on a channel, not a queue. So it always gets one message from any of the queues it subscribes on and does not see the next message unless it deals with that one. (with prefetch 1)

[Chris Duncan]

Hi Vishnu,

On 3 Feb 2010, at 20:01, vishnu wrote:

> Hi Chris
> oh I didnt mean in a multi-threaded model. It looks like the
> java client allows you to subscribe to multiple queues in a single
> thread. It seems to work because a subscription is on a channel,
> not a queue. So it always gets one message from any of the queues
> it subscribes on and does not see the next message unless it deals
> with that one. (with prefetch 1)

Bunny is a simple beast. When you subscribe it starts an infinite
loop with a blocking read in it, so I can't think of a way to do what
you want unless you use a thread per channel. I've not looked at the
Java client in detail, but I suspect that multiple threads would be
needed to do what you describe it doing.

[vishnu]

<forwarding this to the list so it can be indexed or w/e since I had accidentally sent this only to chris>

hi Chris

    ah I see your point. It looks like you would need atleast 2 threads. One to allow people to add subscriptions to multiple queues and one to block on the channel waiting for any message and then matching it with the known subscriptions? How about 2 ways of subscribing to a queue? One which exits immedeatly but allows you to start doing the blocking read after setting all subscriptions? That would allow it to be single threaded.

Something like

queue1.subscribe(:start_now => false) {|msg| puts "message from queue 1 #{msg}"}

queue2.subscribe(:start_now => false){|msg| puts "message from queue 2 #{msg}"}

queue3.subscribe(:start_now => false){|msg| puts "message from queue 3 #{msg}"}    

bunny.start_subscriptions

The main reason I like this is, if I have one subscriber type per queue, this requires a little more work in terms of scaling up by adding x of subscriber 1 and y of subscriber 2. But the only way to avoid that is to put all messages in one queue and have the subscriber differentiate between messages based on content or header information. Ofcourse the api I suggested might be a bit ugly =p