best approaches to use beanstalkd in a multi-threaded app

[OmarShariffDontLikeIt]

HI!

I in the process of writing a multi-threaded client which uses beanstalkd. My question boils down to this: if I have a thread trying to reserve a job from beanstalk, this thread blocks until a job is available. However in other threads I have processing of jobs underway, which require the ability to modify the job in beanstalkd i.e. bury, put in another tube etc; However the beanstalkd connection is blocked, waiting for a new job (which there may not be) so it is impossible for my app to complete the processing of jobs that are currently underway.

Question is: is the best practice to open two connections to beanstalkd e.g. one to handle blocking reserves and another to handle puts, burys etc. or is there an easy way to 'break' the reserve so that I can stop sending reserve requests to the shared connection and complete my job processing before safely shutting down?

I'd appreciate your ideas and thoughts.

Cheers,

Ben

[进中关]

may be you can use reserve-with-timeout, or use two connetions

2013/8/27 OmarShariffDontLikeIt <omarsharif...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beanstalk-tal...@googlegroups.com.
To post to this group, send email to beansta...@googlegroups.com.
Visit this group at http://groups.google.com/group/beanstalk-talk.
For more options, visit https://groups.google.com/groups/opt_out.

[Chad Kouse]

I don't know the best practices on multi-threadfs beanstalkd consumers.  In fact there might not be any -- all of my consumers are single threaded and thus I've never had this problem.

You can't delete a job that another client has reserved so using 2 clients won't work.  Why not encapsulate the whole client into your thread (so - 1 client per thread)?  Otherwise you have an obvious bottleneck built into your multithreaded system (as you are observing.)

-- chad

[OmarShariffDontLikeIt]

"all of my consumers are single threaded and thus I've never had this problem."

Me too. I was hoping to take advantage of multi-threading in a way that meant I was using the connection efficiently.

"You can't delete a job that another client has reserved so using 2 clients won't work."

That would confirm the results of my first dabble at this, which used two connections, though I could of swore I telnetted in from two different sessions and was able to do this.

"Why not encapsulate the whole client into your thread (so - 1 client per thread)?  Otherwise you have an obvious bottleneck built into your multithreaded system (as you are observing.)"

The plan is to provide multiple connections but I'd like each thread to really take advantage of each connection.

At the moment I have a single connection. As each job comes in I keep a track of it. If there are currently active jobs being processed, I switch to reserve-with-timeout, allowing the thread to process results. If there are no active jobs being processed, then the connection switches to traditional blocking reserve. The problem is that the blocking reserve obviously blocks the active thread.

Mmmmm. I may have to sack off the blocking reserve and just go with reserve-with-timeout(0) and poll periodically. Not ideal, but I can't see a way around this at the moment.

Thanks for your help Jerry and Chadkouse.

[Sebastian Lagemann]

Hey,

Am 28.08.2013 um 10:01 schrieb OmarShariffDontLikeIt <omarsharif...@gmail.com>:
>
> At the moment I have a single connection. As each job comes in I keep a track of it. If there are currently active jobs being processed, I switch to reserve-with-timeout, allowing the thread to process results. If there are no active jobs being processed, then the connection switches to traditional blocking reserve. The problem is that the blocking reserve obviously blocks the active thread.
>
> Mmmmm. I may have to sack off the blocking reserve and just go with reserve-with-timeout(0) and poll periodically. Not ideal, but I can't see a way around this at the moment.

You could also use a non-blocking connection which let your thread reading data from the stream as soon as the thread is ready again to read from it.

Best,

Sebastian

[OmarShariffDontLikeIt]

A non-blocking connection? Hmmm interesting. I'm not sure that is an option with the library I am using to connect to beanstalk. Could you expand a little on what you mean by non-blocking connection, and how you envisage this working?

Cheers,

Ben

[abls]

I did develop an async/non-blocking fire-and-forget Nginx module to beanstalkd, it is basically a client but only with put support. All messages are sent trough a non-blocking socket and a separate thread holds an event loop (epoll) to manage POLLHUP/POLLERR events and reconnect the socket based on a retry/polling parameter.

Just drop me a msg if you want any help...

cheers..

[abls]

Just made the module available on https://github.com/arleybls for anyone who wants to store their Nginx logs on a beanstalk server.

cheers..

On Wednesday, August 28, 2013 12:25:25 PM UTC+1, OmarShariffDontLikeIt wrote: