Pool class in PHP

[Andrew Betts]

Hi there,

I suspect I know the answer to this already but I'm a little unsure and would appreciate the clarification.  I'm using beanstalk very successfully, and want to ensure that it delivers high availability.  There didn't seem to be an obvious way to do this and I was intending to adapt my PHP wrapper to try servers in turn when publishing, and to connect and listen to several servers when consuming.  But then I discovered the Pool class, and I found the Ruby implementation of it.  http://search.cpan.org/~gbarr/Beanstalk-Client/lib/Beanstalk/Pool.pm suggests that Perl hasn't got around to implementing it yet and I can't find a PHP implementation.

Looking at the Ruby, it appears that the reserve command is sent to a random server:

def reserve(timeout=nil)
      send_to_rand_conn(:reserve, timeout)
end

So, if that's the case, how would a worker know that a job was available on a server other than the one it sent the reserve to?  Is it simply a matter of running enough workers that there's always at least one watching each beanstalkd server?  Is that the generally recommended way of implementing a pool class (if I were to write one in PHP)?  Would it be better to have each worker connected to all (or at least several) beanstalk servers, and if so, any suggestions for how a single threaded worker should alternate between servers?

And are there any PHP client libraries I'm not aware of that already have this kind of functionality?

Andrew

[Alister Bulman]

On Friday, July 15, 2011 12:19:12 PM UTC+1, Andrew Betts wrote:

So, if that's the case, how would a worker know that a job was available on a server other than the one it sent the reserve to?  Is it simply a matter of running enough workers that there's always at least one watching each beanstalkd server?  Is that the generally recommended way of implementing a pool class (if I were to write one in PHP)?  Would it be better to have each worker connected to all (or at least several) beanstalk servers, and if so, any suggestions for how a single threaded worker should alternate between servers?

And are there any PHP client libraries I'm not aware of that already have this kind of functionality?

There's an open issue for multi-server support in Pheanstalk, but no movement on it yet,
https://github.com/pda/pheanstalk/issues/17

As you'd pretty much said, the two options are to either connect, with a short or no-timeout [reserve(0), in Pheanstalk] to each server in turn, or just be running enough workers and trust that probability will get to them all eventually.

The former means that you could end up in quite a tight loop, so I'd tend towards the latter, as I'll usually be running a dozen or more workers. Since my own queues can also empty out, I'd rather have a few seconds reserve-with-timeout call. Getting stats for the number of ready-jobs from the server is easy enough, so you'll be able to see if the queues are going down or not anyway.

You could also do both as well, choose a couple of beanstalkd servers at random and reserve from one, then try the next, rinse and repeat.

Alister
--
I presented http://abulman.co.uk/presentations/Beanstalkd-2010-05-06/ at PHP London, May 2010

[Chad Kouse]

My typical method is to randomly add jobs to servers (unless for some reason I need things to be consumed in a certain order) and then have multiple consumers each pointed at a different server. 

My tubes all see constant action however so my setup kind of necessitates that. 

--chad

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To view this discussion on the web visit https://groups.google.com/d/msg/beanstalk-talk/-/WZeCNWFau0wJ.
To post to this group, send email to beansta...@googlegroups.com.
To unsubscribe from this group, send email to beanstalk-tal...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

[Andrew Betts]

Thanks Alastair and Chad, that's very helpful.