Managing background workers

[gonzoprosperity]

We have a Rails based app so we use the beanstalk-ruby client.

I am wondering how people have their background workers monitored and
controlled. We usually use God to monitor jobs but I cannot seem to
get it to play nice with the workers.

Our worker code is basically:

@stop = false
trap('TERM') { @stop = true }
loop do
break if @stop
job = queue.reserve
... operate on job...
job.delete
end


Since +reserve+ blocks its possible that you will just sit there
waiting for jobs, thus if a SIGTERM is sent to the script then it wont
actually quit until a job comes along and gets processed - which for
us can be very unpredictable.

My next thought was to specify a reserve timeout

job = queue.reserve(20) # seconds

So that at least every 20 seconds there will be a loop iteration any
SIGTERM sent in the interim can be caught and handled.

But this just doesnt seem very clean.

Whats the methodology that others have successfully employed?

Thanks
/Cody

[ty...@monkeypox.org]

On Fri, 20 Nov 2009, gonzoprosperity wrote:

> My next thought was to specify a reserve timeout
>
> job = queue.reserve(20) # seconds
>
> So that at least every 20 seconds there will be a loop iteration any
> SIGTERM sent in the interim can be caught and handled.

This is the method I've adopted for our Python-based workers, I've basically
got a SIGINT/SIGTERM handler that basically sets "cont = False" and within the
next TIMEOUT period the worker will gracefully die.

Cheers,
-R. Tyler Ballance
--------------------------------------
GitHub: http://github.com/rtyler
Twitter: http://twitter.com/agentdero
Blog: http://unethicalblogger.com

[Al-Faisal El-Dajani]

I have used this "unclean" method in production environments, and it works like a charm.  One thing to keep notice of, if you are using beanstalk ruby client (1.0.2) and your reserve with timeout, there is a leak that will cause the client to keep creating new connection without closing old ones every time it "reserves" and does not find a message.

Check this thread for an explanation of the problem: http://groups.google.com/group/beanstalk-talk/browse_thread/thread/8385631228bf924/2680fb21718972e4?lnk=gst&q=faisal#2680fb21718972e4

And this thread for a solution (p.s: you'll have to patch the gem yourself): http://groups.google.com/group/beanstalk-talk/browse_thread/thread/e8e1176c6dfefc80

--

You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beansta...@googlegroups.com.
To unsubscribe from this group, send email to beanstalk-tal...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=.

--
Al-Faisal El-Dajani
10/6

[Ask Bjørn Hansen]

On Nov 20, 4:05 pm, gonzoprosperity <tool...@gmail.com> wrote:

> I am wondering how people have their background workers monitored and
> controlled. We usually use God to monitor jobs but I cannot seem to
> get it to play nice with the workers.

Graham made a plugin for nagios that we use:

http://search.cpan.org/dist/Nagios-Plugin-Beanstalk/lib/Nagios/Plugin/Beanstalk.pm

Rather than monitor the workers directly, we check with beanstalkd
that there's a worker for each queue and (when appropriate) that the
oldest job isn't older than whatever threshold is appropriate.


- ask

[Steve Farrell]

If you're using rails, and have a full library of gems and plugins, it can be quite expensive to launch individual workers.  What seems to work well is launching a single supervisor process, and forking your workers from that.  I currently do this from a cronjob, but what would be nicer would be to make the supervisor a worker that forks other workers in response to messages.  That approach could be combined with what Ask wrote below, to enable the supervisor to keep an eye on the workers it spawns.