Durable, Ruby-friendly work queues?
Sam Livingston-Gray
I keep hearing people praising Resque, but Igal's rundown[1] indicated
that jobs can't be reserved, and may be dropped on the floor if the
worker dies while performing them. However, the current examples in
the overview page[2] show a worker using a Resque.reserve call. Also,
the README talks about how workers fork to perform work, and suggests
using monit or god to kill stale workers -- which I would *hope* means
that their jobs can be automatically picked up again by another
worker.
I do see that one of Igal's later slides raises a WTF with regard to
lack of durability in Redis, but I note that Redis can be
configured[3] to use the equivalent of a transaction log, so perhaps
this would address the issue.
Igal, was it the Redis durability WTF that led to the Resque WTF, or
was it something else? Or has the issue you saw been addressed in the
intervening time?
Anybody else have a reliable queueing system they'd like to recommend?
-Sam
[1] https://github.com/igal/resque_and_redis_eg/blob/master/resque_and_redis.pdf?raw=true
[2] https://github.com/defunkt/resque
[3] http://redis.io/topics/persistence
Jesse Cooke
--
You received this message because you are subscribed to the Google Groups "pdxruby" group.
To post to this group, send email to pdx...@googlegroups.com.
To unsubscribe from this group, send email to pdxruby+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pdxruby?hl=en.
Igal Koshevoy
> that jobs can't be reserved, and may be dropped on the floor if the
> worker dies while performing them. However, the current examples in
> the overview page[2] show a worker using a Resque.reserve call. Also,
> the README talks about how workers fork to perform work, and suggests
> using monit or god to kill stale workers -- which I would *hope* means
> that their jobs can be automatically picked up again by another
> worker.
means it's only appropriate when you're okay with losing data or have
built another layer on top that's providing the durability.
To see the data loss in action, create a trivial job with a ::perform
method containing "puts 'Hello!'; sleep 60", enqueue the job, start a
worker and let it claim the job, wait until it says "Hello!" and kill
it (CTRL-C or `kill`), and the job's data is gone forever.
Resque uses the word "reserve", but doesn't actually mean it -- i's
really just a "pop".
Here's the Resque 1.16.1 (latest public release) implementation of Job::reserve:
def self.reserve(queue)
return unless payload = Resque.pop(queue)
new(queue, payload)
end
And here's the implementation of Resque::pop
def pop(queue)
decode redis.lpop("queue:#{queue}")
end
The "redis.lpop" tells Redis to return the job and delete it from the
database. So once it's "reserved" (their terminology), it's gone. The
suggestions about using a stale worker killer and nothing else means
they're comfortable with losing any jobs that the stale workers
"reserved".
> I do see that one of Igal's later slides raises a WTF with regard to
> lack of durability in Redis, but I note that Redis can be
> configured[3] to use the equivalent of a transaction log, so perhaps
> this would address the issue.
I haven't used Redis much since 2.0.4. However, I had multiple
complete data losses using releases up to and including that one. E.g.
I'd use Redis, let the code using it finish its activities, let the
Redis server sit idle for a while (e.g. hours), stop the Redis server,
later start it again ... and my database is completely empty. I'm
positive that in at least one of these cases I even issued a Redis
"save" well before stopping the server, which is supposed to force it
to write the data to disk. I'm not sure if the restart cleared the
database, or if the data was never actually written to disk. I see
nothing in the release notes (as of 2.2.7) mentioning fixes for this.
I saw other complains of data loss on the mailing lists, where one
group would say "It lost my data!" and others would reply with "You
lie, that's impossible! It has a transaction log!!1!", so these went
nowhere.
> Igal, was it the Redis durability WTF that led to the Resque WTF, or
> was it something else? Or has the issue you saw been addressed in the
> intervening time?
Both.
> Anybody else have a reliable queueing system they'd like to recommend?
I'd like to hear about this as well. :)
-igal
Sam Livingston-Gray
It seems like one could modify Resque to use an rpoplpush command to
atomically move jobs onto a "work in progress" queue, then add even
yet still more workers to check on those and put them back in the
original queue after a pre-specified timeout... but then I'd worry
about *those* workers, and voila, infinite recursion. ;>
-Sam
Colin Curtin
So we've been dealing with the issue of job queue management for quite
a while on our largish project, which prompted us to take a crack at
solving it "for real" (tm) in "the cloud" (tm). This lets us handle
the durability and consistency of the job queue, as well as metrics
and error handling/alerting.
Ah, Gearman, yet another Danga project that I hope will be taken up en
masse as everyone realizes it's the right solution for this particular
problem, as was the case with memcached. You can read up on it here:
http://en.wikipedia.org/wiki/Gearman and
http://danga.com/words/2007_06_usenix/usenix.pdf (probably the best
slide deck I've found regarding scaling ever. The gearman part starts
on page/slide 83. Why the best? Because Brad takes you through the
entire LJ stack's history and breaking points.)
We're in private alpha right now and getting some really good
feedback. Sign up for the list, and if you have urgent need, please
email me separately.
OH! And one more thing: gearmand (the c impl) is open source and is
what we've been using up until now. It has its own set of issues, but
it's fast, stable, language agnostic. We're the maintainers of the
Ruby Gearman lib at https://github.com/gearman-ruby/gearman-ruby,
which we'll be updating with more worker-centric code soon.
Thanks,
Colin
> --
> You received this message because you are subscribed to the Google Groups "pdxruby" group.
> To post to this group, send email to pdx...@googlegroups.com.
> To unsubscribe from this group, send email to pdxruby+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/pdxruby?hl=en.
>
>
--
=begin
Colin Curtin
Chief Happiness Officer, Cramer Development
http://cramerdev.com
email: co...@cramerdev.com
skype: colin.t.curtin
phone: +1.805.694.UNIX (8649)
=end
Igal Koshevoy
> It seems like one could modify Resque to use an rpoplpush command to
> atomically move jobs onto a "work in progress" queue, then add even
> yet still more workers to check on those and put them back in the
> original queue after a pre-specified timeout... but then I'd worry
> about *those* workers, and voila, infinite recursion. ;>
That's not as silly as you make it sound. The solution I was thinking
about implementing, before I got fed up with the Redis data loss was:
* Change workers to use Redis' atomic "rpoplpush" to reserve jobs by
moving them from the original queue into a reserved pool.
* Change workers to push a job id into a completed queue when done.
* Create a nanny daemon that either uses polling or subscriptions to
watch the reserved pool and completed queue.
* Nanny reads the completed queue and removes any matching jobs from
the reserved pool, because they're done.
* Nanny adds a timestamp to any new job in the reserved queue that
doesn't already have a timestamp, since it's new.
* Nanny searches for jobs in the reserved pool whose timestamps
indicate they've timed out, and re-enqueues them.
* Use monit/god/whatever to make sure the nanny is running.
* Pretend that monit/god/whatever never crashes to avoid implementing
an infinite recursion of watchers. :)
Other than the "pretend" bit at the end, does this make sense as a solution?
If Redis' data loss issues have been resolved, this would be worth
building. Maybe we can throw together a prototype at the Thursday
hackathon? :) http://calagator.org/events/1250460452
-igal
Ezra Zygmuntowicz
Rescue could easily be made to not lose data but it does not take advantage of the proper redis commands. I begged Salvatore to add a blpoprpush command for this verty reason. This command does a blocking pop against a certain lists, when it gets an item in the list, it also places a copy of that item in the secondary list or thge "in flight" list. This way if the worker crashes you have another daemon that scans tyhe "in flight" queue every so often and puts work itens back in the main queue if they have sat for too long.
In pseudocode:
workitem = blpoprpush 'work-queue', 'work-queue-inflight'
# working on item....
# done working on work item so I LREM the qwork item from 'work-queue-inflight' and all is well.
# if I were to crash while working on an item, I would leave it orphaned in the 'work-queue-inflight' queue and
# the garbage collector daemon would come along and notice that my work item had been sitting in 'work-queue-inflight' for longer then
# whatever timeout has been set for work items.
This is the pattern that the main loop of resque should use. This emulates AMQP's acknowledgement system and makes it so you never lose work items, but they may be run twice so should be idempotent.
That is kindof the tradeoff with qworker systems. If you want them to be truly never lose work then you mustr be ready to accept the fact that work items may be processed twice and have code to accomodate for that. Or you can have it so work will only every get processed once but run the risk of some work items getting dropped on the floor and never processed.
Cheers-
Ezra Zygmuntowicz
ezmo...@gmail.com
Igal Koshevoy
> YAY I get to plug something I've been working on!
>
> http://gearmanhq.com/
>
> So we've been dealing with the issue of job queue management for quite
> a while on our largish project, which prompted us to take a crack at
> solving it "for real" (tm) in "the cloud" (tm). This lets us handle
> the durability and consistency of the job queue, as well as metrics
> and error handling/alerting.
>
> Ah, Gearman, yet another Danga project that I hope will be taken up en
> masse as everyone realizes it's the right solution for this particular
> problem, as was the case with memcached. You can read up on it here:
> http://en.wikipedia.org/wiki/Gearman and
> http://danga.com/words/2007_06_usenix/usenix.pdf (probably the best
> slide deck I've found regarding scaling ever. The gearman part starts
> on page/slide 83. Why the best? Because Brad takes you through the
> entire LJ stack's history and breaking points.)
>
> We're in private alpha right now and getting some really good
> feedback. Sign up for the list, and if you have urgent need, please
> email me separately.
>
> OH! And one more thing: gearmand (the c impl) is open source and is
> what we've been using up until now. It has its own set of issues, but
> it's fast, stable, language agnostic. We're the maintainers of the
> Ruby Gearman lib at https://github.com/gearman-ruby/gearman-ruby,
> which we'll be updating with more worker-centric code soon.
Gearman at a Perl meeting a while ago, and it had sensible API.
However, it only had in-memory queues at the time, which limited its
usefulness.
On initial reading, it sounds like the newly-added persistent queuing
and retry system makes sense, and supports Drizzle, MySQL, PostgreSQL
and sqlite3 databases:
http://gearman.org/index.php?id=manual:job_server#persistent_queues
Have folks used this? Thoughts? Experiences?
Also, there's two talks at Open Source Bridge that may be of interest:
* "Qs on Queues" is an overview of queuing systems by Eric Day, one of
the co-authors of Gearman: http://opensourcebridge.org/sessions/656
* "Gearman: From the Worker's Perspective" is a talk by Brian Aker,
one of the co-authors of Gearman and major contributor to MySQL and
such: http://opensourcebridge.org/sessions/700
-igal
Sam Livingston-Gray
> That's not as silly as you make it sound. The solution I was thinking
> about implementing, before I got fed up with the Redis data loss was:
> * Change workers to use Redis' atomic "rpoplpush" to reserve jobs by
> moving them from the original queue into a reserved pool.
> * Change workers to push a job id into a completed queue when done.
> * Create a nanny daemon that either uses polling or subscriptions to
> watch the reserved pool and completed queue.
> * Nanny reads the completed queue and removes any matching jobs from
> the reserved pool, because they're done.
> * Nanny adds a timestamp to any new job in the reserved queue that
> doesn't already have a timestamp, since it's new.
> * Nanny searches for jobs in the reserved pool whose timestamps
> indicate they've timed out, and re-enqueues them.
> * Use monit/god/whatever to make sure the nanny is running.
> * Pretend that monit/god/whatever never crashes to avoid implementing
> an infinite recursion of watchers. :)
>
> Other than the "pretend" bit at the end, does this make sense as a solution?
I was wondering if you can remove items from an arbitrary point in a
Redis list, but then I RTFMd and see that the LREM command does just
that. So, yeah, I think that would work.
> If Redis' data loss issues have been resolved, this would be worth
> building. Maybe we can throw together a prototype at the Thursday
> hackathon? :) http://calagator.org/events/1250460452
I'm busy with a production deploy this Thursday (I do one of these
every fourth Thursday), but I can often arrange an evening out with a
bit of advance notice -- especially if I do it before my partner plans
meals for the week. (=
Also, my company is moving offices at the end of the month (we'll be
across the street from NewRelic), and we've been talking about making
our space available for community hacking events. This kind of thing
would be especially nice, since it could benefit us directly, but
direct self-interest is not a requirement for us to share the space.
More on this in June, I hope.
-Sam
Matthew Boeh
> building. Maybe we can throw together a prototype at the Thursday
> hackathon? :) http://calagator.org/events/1250460452
I'd be interested. I'm pretty satisfied with RabbitMQ right now but
it'd be nice to have an alternative in mind.
Matthew Boeh
Igal Koshevoy
Ezra,
Thanks for the confirmation that what I'd describe would work.
If I have time, I'll see if I can rework Resque to detect whether the
Redis database it's talking to supports the blpoprpush command and use
it if possible, because it currently just does polling, although
that's not as much of a worry given how quick Redis is.
-igal
Igal Koshevoy
I'm going to wander over to the Lucky Lab for tonight's hackathon.
I'll take a shot at doing what I described above with the current
versions of Resque and Redis. If anyone feels interested, drop on by
whenever. I'll post an update later.
-igal
Igal Koshevoy
> I'm going to wander over to the Lucky Lab for tonight's hackathon.
> I'll take a shot at doing what I described above with the current
> versions of Resque and Redis. If anyone feels interested, drop on by
> whenever. I'll post an update later.
I worked on this the other night at the hackathon with my awesome
ex-coworker Nick Lewis. The reservation and retry mechanism turned out
to be more complex than I'd thought earlier. I've written a fairly
complete spec for implementing this functionality.
However, after I finished this specification, I found a relevant
conversation in the Resque issues tracker. The author of Resque
absolutely doesn't want it to reserve or retry, and has rejected
patches for this: "Resque does not advertise itself as a system that
will never lose your jobs. Part of the design is we don't care if jobs
are lost."
Given that such patches will never be accepted, this is probably not
the way to go.
You'll see my findings and proposed solution at https://gist.github.com/983749
-igal
Sam Livingston-Gray
> However, after I finished this specification, I found a relevant
> conversation in the Resque issues tracker. The author of Resque
> absolutely doesn't want it to reserve or retry, and has rejected
> patches for this: "Resque does not advertise itself as a system that
> will never lose your jobs. Part of the design is we don't care if jobs
> are lost."
That's... really special. I'd suggest that an appropriate response
would be "fork you," but that's probably not even worthwhile.
Jesse Cooke
Ben Bleything
> However, after I finished this specification, I found a relevant
> conversation in the Resque issues tracker. The author of Resque
> absolutely doesn't want it to reserve or retry, and has rejected
> patches for this: "Resque does not advertise itself as a system that
> will never lose your jobs. Part of the design is we don't care if jobs
> are lost."
This is exactly why we use DJ. Our async jobs rely on flaky external
services and fail regularly. The plugins and whatnot for retry in
resque are crap and resque itself will never support it so... yeah.
It's too bad too, because DJ doesn't really handle multiple workers
and high job volume well. We waste a tremendous amount of time in lock
contention with just 12 workers.
Allegedly there's a redis backend for DJ but I haven't heard of anyone
using it. I've heard some good things about the Mongo backend but I'm
not deploying mongo to support my job system.
I think there's space for a good alternative to DJ and Resque that
still uses redis, but I don't have time to build it :(
Ben
Jesse Cooke
Ben Bleything
> Like I said, been happy with mongo as a backend for DJ, and it's as easy as
> redis to deploy.
Sure. To be clear, I've already got redis in production, so it'd be
nice to just use what I've got.
Ben
Sam Livingston-Gray
On Tue, May 17, 2011 at 3:13 PM, Igal Koshevoy <ig...@pragmaticraft.com> wrote:
> I haven't used Redis much since 2.0.4. However, I had multiple
> complete data losses using releases up to and including that one. E.g.
> I'd use Redis, let the code using it finish its activities, let the
> Redis server sit idle for a while (e.g. hours), stop the Redis server,
> later start it again ... and my database is completely empty. I'm
> positive that in at least one of these cases I even issued a Redis
> "save" well before stopping the server, which is supposed to force it
> to write the data to disk. I'm not sure if the restart cleared the
> database, or if the data was never actually written to disk.
I wonder if you ran afoul of this issue:
http://groups.google.com/group/redis-db/msg/1703490e2b2a3354
tl;dr Redis forks a child process to write to disk, and if Linux
doesn't have enough free RAM to duplicate all the pages from the
parent, the child process will silently die. (Nice, huh?)
This is addressed at http://redis.io/topics/faq -- search for
"overcommit_memory".
-Sam
Igal Koshevoy
Thanks for the research.
I'd be surprised if I was out of memory because I use
"overcommit_memory" mode and would have expected other programs to
have failed at the same time if there was an issue. The only listed
cause that could have been a potential culprit was, "It seems to me I
have accidentally run two redis instances and the new one emptied the
dump.rdb". What a ridiculous failure mode -- maybe I did this too
somehow, like running "redis-server -h" to see what arguments it took,
but because it doesn't honor the "-h" or "--help" flags like any
reasonable UNIX command should, it might have instead tried to start
and nuked my database. Sigh.
Also, earlier this month ep.io (like Heroku for Python apps) had a
high-profile Redis failure, where it stopped writing changes to disk,
and then wiped their database file clean on restart:
http://www.ep.io/blog/todays-outage/
So adding durability to Resque may not be worth doing if Redis itself
isn't durable.
Maybe the "right" approach is to accept that Resque and Redis are
fast, but unreliable systems for unimportant data -- and that you
should use something like DelayedJob and PostgreSQL for any data you
actually want to keep. Using both of these on the same project should
be fine.
-igal