Multiple observers, single consumer...
72 views
Skip to first unread message
DeRailed
Apr 29, 2009, 4:58:48 PM4/29/09
to beanstalk-talk
Hi,
First of all thanks for this great project !
I have a queue with a series of observers that just look at jobs as
they come by and release then right
away and a single consumer that will actually delete the job from
the queue.
I would like to implement a strategy where all "observers" get a
crack at looking at the job before it is
actually consumed by the single consumer. Hence having the single
consumer give all other workers a
chance to look at the job before it actually consumes it.
Is this feasible with beanstalkd ? If so how ? I am currently using
the ruby beanstalk-client to interface with
the queue.
Thanks !!
First of all thanks for this great project !
I have a queue with a series of observers that just look at jobs as
they come by and release then right
away and a single consumer that will actually delete the job from
the queue.
I would like to implement a strategy where all "observers" get a
crack at looking at the job before it is
actually consumed by the single consumer. Hence having the single
consumer give all other workers a
chance to look at the job before it actually consumes it.
Is this feasible with beanstalkd ? If so how ? I am currently using
the ruby beanstalk-client to interface with
the queue.
Thanks !!
Erich
Apr 30, 2009, 1:21:43 PM4/30/09
to beanstalk-talk
Hi,
A few questions about your situation:
Does order matter amongst the observers?
Are all the observers potential consumers?
How is the decision made about consuming (e.g. can there be 2 possible
consumers and how do you pick?)
Can your situation be better handled by an MQ like RabbitMQ for the
broadcast and a job queue (beanstalkd) for the "only one consumer"
aspect?
Regards,
Erich
A few questions about your situation:
Does order matter amongst the observers?
Are all the observers potential consumers?
How is the decision made about consuming (e.g. can there be 2 possible
consumers and how do you pick?)
Can your situation be better handled by an MQ like RabbitMQ for the
broadcast and a job queue (beanstalkd) for the "only one consumer"
aspect?
Regards,
Erich
DeRailed
Apr 30, 2009, 5:24:44 PM4/30/09
to beanstalk-talk
Hi Erich,
Thanks for the reply ! Please see annotations below.
Best Regards...
On Apr 30, 11:21 am, Erich <sophac...@gmail.com> wrote:
> Hi,
>
> A few questions about your situation:
>
> Does order matter amongst the observers?
>
No - Any order is fine as long as the get to see that job
> Are all the observers potential consumers?
No - All observers are just that. They will look at the job do their
thing and push the job back on the queue
>
> How is the decision made about consuming (e.g. can there be 2 possible
> consumers and how do you pick?)
In my scenario, there would only be a single consumer at any time in
the system. Only he can delete
the job
>
> Can your situation be better handled by an MQ like RabbitMQ for the
> broadcast and a job queue (beanstalkd) for the "only one consumer"
> aspect?
>
I haven't look at RabbitMQ as of yet, was hoping this
scenario could be handled with a single q ? Do you see any ways
to make this work ?
Thanks for the reply ! Please see annotations below.
Best Regards...
On Apr 30, 11:21 am, Erich <sophac...@gmail.com> wrote:
> Hi,
>
> A few questions about your situation:
>
> Does order matter amongst the observers?
>
> Are all the observers potential consumers?
thing and push the job back on the queue
>
> How is the decision made about consuming (e.g. can there be 2 possible
> consumers and how do you pick?)
the system. Only he can delete
the job
>
> Can your situation be better handled by an MQ like RabbitMQ for the
> broadcast and a job queue (beanstalkd) for the "only one consumer"
> aspect?
>
scenario could be handled with a single q ? Do you see any ways
to make this work ?
Dan Mayer
Apr 30, 2009, 7:59:12 PM4/30/09
to beansta...@googlegroups.com
Just some thoughts that might help...
If you wanted you could just put the message onto multiple tubes
(queues)... Each listener would just pull it off their tube and delete
it. The consumer would do the same. Occasionally this is the simplest
way to broadcast over beanstalk. It puts more messages through the
system but if you aren't expecting it to be super high load this might
be fine.
In our situation we found that having duplicate tubes for all the
listeners to work just fine. Then we just have a broadcast all method
that puts a message onto all the listening queues in the system.
Off the top of my head that is the best way to do it with beanstalk,
but I would like to see what ever other thoughts people have on this.
peace,
Dan
If you wanted you could just put the message onto multiple tubes
(queues)... Each listener would just pull it off their tube and delete
it. The consumer would do the same. Occasionally this is the simplest
way to broadcast over beanstalk. It puts more messages through the
system but if you aren't expecting it to be super high load this might
be fine.
In our situation we found that having duplicate tubes for all the
listeners to work just fine. Then we just have a broadcast all method
that puts a message onto all the listening queues in the system.
Off the top of my head that is the best way to do it with beanstalk,
but I would like to see what ever other thoughts people have on this.
peace,
Dan
Keith Rarick
May 1, 2009, 5:22:52 AM5/1/09
to beansta...@googlegroups.com
One other possible approach:
* Make two tubes, S and T.
* The observers listen on S, the consumer listens on T.
* Put each job into S.
* When an observer gets a job, it can copy the job to T and delete
the original from S.
This lets exactly one observer see each job before it moves on to the consumer.
If you give more details about your situation we might be able to
offer more (or better) suggestions.
kr
* Make two tubes, S and T.
* The observers listen on S, the consumer listens on T.
* Put each job into S.
* When an observer gets a job, it can copy the job to T and delete
the original from S.
This lets exactly one observer see each job before it moves on to the consumer.
If you give more details about your situation we might be able to
offer more (or better) suggestions.
kr
Erich
May 1, 2009, 8:42:00 AM5/1/09
to beanstalk-talk
I understand your desire to keep as much of the functionality as
possible within a few programs. Keith and Dan provide solutions below
that are what I would provide too. If this is a webapp, a 3rd
approach would be to have your observer grab the job from a webserver
in serialized xml/json/whatever format, but that would be a pretty
serious restructuring.
The reason I suggested RabbitMQ as a potential partial solution is
semantics, the short is beanstalkd has an anycast semantic while
rabbitmq has a multicast semantic. There is a thread a while back
about Second Life's MQ ratings, in which i spend way too many words on
this subject.
Good luck,
Erich
possible within a few programs. Keith and Dan provide solutions below
that are what I would provide too. If this is a webapp, a 3rd
approach would be to have your observer grab the job from a webserver
in serialized xml/json/whatever format, but that would be a pretty
serious restructuring.
The reason I suggested RabbitMQ as a potential partial solution is
semantics, the short is beanstalkd has an anycast semantic while
rabbitmq has a multicast semantic. There is a thread a while back
about Second Life's MQ ratings, in which i spend way too many words on
this subject.
Good luck,
Erich
Jeremy Dunck
May 1, 2009, 9:16:15 AM5/1/09
to beansta...@googlegroups.com
On Fri, May 1, 2009 at 7:42 AM, Erich <soph...@gmail.com> wrote:
...
>
> There is a thread a while back
> about Second Life's MQ ratings, in which i spend way too many words on
> this subject.
...
>
> There is a thread a while back
> about Second Life's MQ ratings, in which i spend way too many words on
> this subject.
FWIW, I enjoyed that message. We didn't cover various queue
strategies in my CS program. :-/
Erich
May 1, 2009, 10:08:27 AM5/1/09
to beanstalk-talk
On May 1, 8:16 am, Jeremy Dunck <jdu...@gmail.com> wrote:
> FWIW, I enjoyed that message. We didn't cover various queue
> strategies in my CS program. :-/
particularly the paper on "brokers vs brokerless". It gets into higher
level architectures and concepts pretty well.
Regards,
Erich
DeRailed
May 2, 2009, 11:43:37 AM5/2/09
to beanstalk-talk
Thank you all for you feedback !
Dan - I think the issue, in my case, with your solution is that the
producer will have to be aware
of all the consumers. ie it will have to create tube accordingly. I
should have mentioned this
in my original post, my bad !
Keith - Perhaps I am missing something. I thought about creating 2
tubes, one for the true observers
and another for the consumer, but I think ??, this raises the same
issue as the single tube approach,
since one of the observer will have to know how to publish an event to
the consumer tube. The issue
there and perhaps my misunderstanding, is when and who should consume
the messages from the
observer tube. Seems to be the same issue since that observers can not
guarantee that all other
observers have seen that message.
Erich - I will take a look a the second life thread
So I think there might be 2 approaches here:
A) Somehow track all the observers and record the fact that a message
was seen by a given observer.
Then the observer that pick it up last deletes the message. (I think
that's lame?)
B) Have someway to expire a message. ie if a message is in a queue for
a greater time then x then flush it
out. Not sure how one would achieve this ?
So to recap my situation
o The producer has no idea how many workers there will be
o A job should be deleted from the queue once all observers had a
chance to see it.
o Each observers has no idea who else is listening on the queue.
Thanks !
Dan - I think the issue, in my case, with your solution is that the
producer will have to be aware
of all the consumers. ie it will have to create tube accordingly. I
should have mentioned this
in my original post, my bad !
Keith - Perhaps I am missing something. I thought about creating 2
tubes, one for the true observers
and another for the consumer, but I think ??, this raises the same
issue as the single tube approach,
since one of the observer will have to know how to publish an event to
the consumer tube. The issue
there and perhaps my misunderstanding, is when and who should consume
the messages from the
observer tube. Seems to be the same issue since that observers can not
guarantee that all other
observers have seen that message.
Erich - I will take a look a the second life thread
So I think there might be 2 approaches here:
A) Somehow track all the observers and record the fact that a message
was seen by a given observer.
Then the observer that pick it up last deletes the message. (I think
that's lame?)
B) Have someway to expire a message. ie if a message is in a queue for
a greater time then x then flush it
out. Not sure how one would achieve this ?
So to recap my situation
o The producer has no idea how many workers there will be
o A job should be deleted from the queue once all observers had a
chance to see it.
o Each observers has no idea who else is listening on the queue.
Thanks !
Keith Rarick
May 4, 2009, 6:19:44 AM5/4/09
to beansta...@googlegroups.com
On Sat, May 2, 2009 at 8:43 AM, DeRailed <fernand...@gmail.com> wrote:
> Seems to be the same issue since that observers can not
> guarantee that all other
> observers have seen that message.
> Seems to be the same issue since that observers can not
> guarantee that all other
> observers have seen that message.
Sorry, I misunderstood. I thought you wanted at least one observer to
see each message, but you want all observers to get each message.
You can get what you want on top of beanstalkd, but it will be a hack.
You really want a message bus; beanstalkd just isn't designed for that
situation. Maybe some day if its features continue creeping, but not
right now.
kr
Reply all
Reply to author
Forward
0 new messages