Retry count strategies?

[Parand Darugar]

I have a scenario where I need to implement retries for jobs in the
queue - try the job three times, and on the 4th try put it into the
failed queue. What's a good way to handle this? Is there a place to
attach meta-data (eg. number of retries) on the job itself, or do I
have to modify the payload of the job and resubmit it?

This is using the beanstalkc python library.

[Dan Mayer]

I think you can do this by viewing the jobs stats (my example is in
Ruby but python should work the same.)

job = beanstalk.reserve
p job.stats
{"releases"=>1, "delay"=>0, "kicks"=>0, "buries"=>0, "id"=>10,
"tube"=>"api-run-requests", "time-left"=>115, "ttr"=>120,
"pri"=>65536, "timeouts"=>1, "age"=>143, "state"=>"reserved"}

From the stats you can view releases and timeouts. If you are
explicitly doing a release you can just check the count of
job['releases'] and if it is less than 3 try it, if it is more than 3
just delete the job. If the jobs are failing in a way that they are
put back on the queue because of the TTR timeout the timeout will
increase. I think you might want something like

if (job['releases'].to_i + job['timeouts'].to_i) < 3
do_work(job)
else
job.delete
end

We have done this in some code. Another option that we did once was to
include a field called failed on our job objects. When we knew we
couldn't complete a job we incremented the failed count and put it
back on the queue. I think it is better to use the releases and
timeouts provided by beanstalkd. But if you want to control what is
considered a failed attempt this other way can work too.

Hope that helps.

peace,
Dan Mayer
Co-founder, Devver.net (http://devver.net)
http://twitter.com/devver
http://mayerdan.com
http://twitter.com/danmayer

[Parand Darugar]

Thanks Dan, the release count strategy makes a lot of sense, we'll
give that a try.

Best,

Parand

> Co-founder, Devver.net (http://devver.net)http://twitter.com/devverhttp://mayerdan.comhttp://twitter.com/danmayer

[Andrew Gomilko]

Hi Parand,

You can use 'releases' value got from job stats which shows number of
released performed on the given job.
My scenario is: for every job fail, release with incremented delay.

while True:
job = inQueue.reserve()
try:
performUsefulStuff(job)
except:
releases = int(job.stats()['releases'])
job.release(delay=60*releases)

[Martyn Loughran]

If you want to retry jobs it's worth looking at the ruby Job#decay
method which exponentially increases the delay and buries a job once a
delay limit is exceeded.

def decay(d=([1, delay].max * 1.3).ceil)
return bury() if delay >= DELAY_MAX
release(pri, d)
end

Martyn

2009/11/9 Andrew Gomilko <andrew....@gmail.com>:

[Andrew Betts]

In some circumstances it might be quite inefficient to have to call stats-job before a release to find out the job's release history.  I suspect the answer to this is no, but is there any way of having such stats included in the RESERVE response?

[Keith Rarick]

On Sat, Aug 20, 2011 at 5:42 AM, Andrew Betts <andrew...@gmail.com> wrote:
> In some circumstances it might be quite inefficient to have to call
> stats-job before a release to find out the job's release history.

There's no point in speculating about hypothetical performance
problems. If someone encounters a situation where having to call
stats-job causes trouble, we can think about how to fix it.

kr

[Graham Barr]

We currently have a system that is doing about 50 jobs/sec and every one
does a stats-job after the reserve and we do not see a performance issue

Graham.

[Andrew Betts]

Keith - notice I was not requesting such a feature, only checking that it did not exist, which I'll continue to assume is the case.  In our use case, I only need the stats on jobs that need requeuing because the worker has failed to process it in some way, so we only call stats-job on a small fraction of jobs.  I have experienced no performance issues when using Beanstalk but I would simply prefer to set it up in an optimal way.

Thanks for the response.

[Keith Rarick]

On Tue, Aug 23, 2011 at 3:13 AM, Andrew Betts <andrew...@gmail.com> wrote:
> Keith - notice I was not requesting such a feature, only checking that it
> did not exist, which I'll continue to assume is the case.

My apologies. You're right, the server doesn't include statistics in
a RESERVED response.

kr