Moving Forward - A Canonical Repo
dstrelau
As of today, tobi's delayed_job[1] has 108 forks on GitHub. While this
means
there is certainly a whole bunch of features, bug fixes and just plain
awesome,
I frankly have no idea where to look for any of that. With so many
forks, I
wouldn't be suprised to find that no two people are using the same
version of
DJ.
With your help, I'd like to fix this. Here are the first steps I'd
like to
propose to get this sorted out:
- Setup a canonical GitHub repository (delayedjob/delayed_job) and
give commit
access to those who have made significant improvements already. This
would
become _the place_ to file issues and make pull requests to. It
would also be
the repository from which the delayed_job gem is created.
- Figure out what fork is currently considered the "best" starting
point for
future work. It seems that both collectiveidea[1] and weplay[2] have
done a
fair amount of work, but there are probably others I have
overlooked. Does it
seem like a good idea to start from tobi's repo and merge others in
after
review or should we start with something more recent?
- Consolidate. Gradually bring in commits that are considered
"essential" to
produce what would essentially be v2.0. Then, consolidate bugs/
features from
all the forks and create a master issue list. Ideally, bring over
any wiki
documentation into the canonical version as well.
Obviously, this is not an overnight project, but it would be very
beneficial to
the long-term maintenance of the project. While GitHub forks are
great, they
become chaotic if there is no "blessed" repository that actually
merges things
back in. Having a central repository will allow users to more easily
keep track
of bugfixes and enhancements and upgrade to newer versions.
Please let me know your thoughts. I will gladly organize the effort to
make this
happen, but I want to work with as many people as are interested in
helping.
Likewise, if this all seems like a terrible idea, please speak up!
Thanks,
Dean Strelau
[1]: <http://github.com/tobi/delayed_job>
[2]: <http://github.com/collectiveidea/delayed_job>
[3]: <http://github.com/weplay/delayed_job>
Jerod Santo
Tobi
Having started the project i'd like to communicate some of my thoughts
about the project in this space though:
Delayed Job isn't as good as people think. It's convenient and it
works, this puts it above a lot of competing projects but it's still
not great. The biggest issue is my mistake of adding priority to the
project. I suggest scrapping this feature.
The reason is that rails cannot easily create indexes that accelerate
"DJ's aquire new jobs" query. This is partly because of inverted sort
orders. The standard query looks for jobs order by run_at ASC,
priority DESC. Rails has no way of creating such an index that inverts
on the second column so you have to drop to straight SQL for it.
However, even if you have this index DJ is still a ticking time bomb.
For example our www stats log server creates around 1-2m jobs a day.
If anything goes wrong with a job worker, even for an hour it will
make the server collapse under it's own weight because the select
statement can easily take longer to return then new jobs come in which
ends in a positive feedback loop.
Lastly the job lock queries cause table locks on all mysql versions
below 5.1. This leads multiple workers to degenerate performance
immensely.
Here is how I think DJ should go forward:
* Scrap priorities, maybe replace them with named queries which are
easier to index. This would allow people to create a high priority
bucket with dedicated job workers.
* Include a daemon that works well in an unix environment. Look at
unicorn for inspiration. E.g. there should be one master process which
aquires new jobs from the DB and then sends them through pipes to the
job workers round robin style. This will help immensely with scaling
out workers for higher work loads
* Involve some real SQL gurus with the project and optimize the heck
out of it. Make sure everything hits the right indexes and nothing
causes locks
* Add a new table to DJ that can act as a log. This means that we can
again - by default - delete jobs once they are done from the DB. For
each finalized job there should be a record. Multiple records should
be collpsed into hourly granualarity. Then ship with a standard
controller (engine?) that allows people to get some visibility into
how their DJ is performing. Jobs/second processed vs incoming etc.
* Ship with a plugin for collectd and similar, but be very careful
with select COUNT(*) calls because they can bring down the server with
many million of rows on innodb ( Shopify's stats server was down
because of this last week)
* Introduce named jobs that allow friendly UI to users of your
website, informing them what's going on at the moment ( we got this in
Shopify already )
Thanks for keeping this project alive. I'm thrilled with the traction
it has gotten. I just hate to see so many people use it without
knowing that there are true scaling problems in it.
On Oct 8, 6:02 pm, Jerod Santo <jerod.sa...@gmail.com> wrote:
> It sounds like a good idea. I also found the inactivity of tobi's repository
> & the number of active/semi-active forks overwhelming.
>
Brandon Keepers
Thanks for taking the initiative on this.
On Oct 8, 2009, at 5:51 PM, dstrelau wrote:
> - Setup a canonical GitHub repository (delayedjob/delayed_job) and
> give commit
> access to those who have made significant improvements already. This
> would
> become _the place_ to file issues and make pull requests to. It
> would also be
> the repository from which the delayed_job gem is created.
It'd be ideal if tobi could just add committer to his repo. But if he
doesn't want to be involved, maybe a separate account is the best
option. Although I'd prefer that the account be called "delayed_job"
to keep the naming consistent.
> - Figure out what fork is currently considered the "best" starting
> point for
> future work. It seems that both collectiveidea[1] and weplay[2] have
> done a
> fair amount of work, but there are probably others I have
> overlooked. Does it
> seem like a good idea to start from tobi's repo and merge others in
> after
> review or should we start with something more recent?
We've tried to be intentional about pulling in the stuff we liked into
the collectiveidea fork. I'd be up for using that as a starting
point, but I'm obviously biased. :)
> Obviously, this is not an overnight project, but it would be very
> beneficial to
> the long-term maintenance of the project. While GitHub forks are
> great, they
> become chaotic if there is no "blessed" repository that actually
> merges things
> back in. Having a central repository will allow users to more easily
> keep track
> of bugfixes and enhancements and upgrade to newer versions.
> Please let me know your thoughts. I will gladly organize the effort to
> make this
> happen, but I want to work with as many people as are interested in
> helping.
> Likewise, if this all seems like a terrible idea, please speak up!
I'd be happy to help you spearhead this. I own the gem on
gemcutter.org, and the mailing list, so I can add whoever needs access
to those.
Thanks,
Brandon
Brandon Keepers
First, thanks for writing delayed_job, and second, thanks for joining
this conversation.
On Oct 9, 2009, at 1:43 PM, Tobi wrote:
> I support this idea.
>
> Having started the project i'd like to communicate some of my thoughts
> about the project in this space though:
>
> Delayed Job isn't as good as people think. It's convenient and it
> works, this puts it above a lot of competing projects but it's still
> not great. The biggest issue is my mistake of adding priority to the
> project. I suggest scrapping this feature.
The great thing about delayed_job is that it is easy for small to
medium sized projects. There are plenty appropriate queues for large
scale apps, but almost nothing for simple background processing on
smaller apps.
Personally, I would prefer to see delayed_job (or a fork of it) be
intentional about not sacrificing simplicity in the interest of
scaling. If we can solve all of the issues you outlined without
making that sacrifice, then I'm on board!
Thanks,
Brandon
Dean Strelau
I think so too. Tobi, is this something you'd be up for or do you
prefer to keep your own fork your own? (And just out of curiosity, is
Shopify still using tobi/delayed_job as is or have you made more
modifications?)
> We've tried to be intentional about pulling in the stuff we liked into the
> collectiveidea fork. I'd be up for using that as a starting point, but I'm
> obviously biased. :)
> I'd be happy to help you spearhead this. I own the gem on
> gemcutter.org, and the mailing list, so I can add whoever needs
> access to those.
Awesome. The collectiveidea fork looks like it has lots of good
changes, so I am up for starting from there too. I didn't realize you
owned the gem and list as well. Sounds like you've been trying to keep
dj organized by yourself so far. Maybe I will be the one helping you
:)
> The great thing about delayed_job is that it is easy for small to medium
> sized projects. There are plenty appropriate queues for large scale apps,
> but almost nothing for simple background processing on smaller apps.
Agreed. Maybe the delayed_job docs just need to be more explicit in
explaining the "there be dragons" factor.
I appreciate the feedback, everyone. This afternoon I'm going to look
at all the forks (hooray 'gh network fetch') and see if I can't
organize a list of what changes everyone has contributed. I'll report
back once I have a better idea of where everything stands.
Dean
BrianTheCoder
rewrote it to use DataMapper instead of AR. I love the idea of a
unicorn style worker, that'd blow my mind. Another thing I've been
thinking about is memory consumption. Delayed Job loads the whole
rails env, which is convient. However it leads to a large amount of
memory bloat when all it really needs is the models. I mean why load
ActionPack into memory when its not used? It might even be worth
splitting the store away from the worker. That way people could easily
write adapters for the latest and greatest storage system to come out
(Tokyo, Mongo, Couch, etc.). That way its a more flexible jobbing
system, plus you may see an advantage using one data store for your
app and one for your jobbing queue depending on performance
implications.
Tobias Lütke
Frankly this failed miserably when I brought in people to the typo
project.
I haven't had a chance to check out the 100+ forks on GitHub so it
would be great if someone could summarize the major advantages of some
of the third party work. I tend to decide against adding features.
Unless a new feature makes every users life better it will have
subtracted value from the project as a whole. In other words I only
want to add things that "most of the people need most of the time".
That's the philosophy behind all the software I write. If I give out
commit keys I can no longer enforce this.
I already said that I would take DJ in a different direction if I had
time to work on it. Unfortunately the success of Shopify means that I
don't get to do a lot of coding anymore. I would make DJ a lot simpler
by removing priorities and focusing on a grade A daemon runner. I
would also make DJ self migrating so that on the first use it will
create the necessary tables or apply table modifications from an old
version so that users don't have to worry about this stuff. I highly
doubt that I would add a single feature to the code base form the
state it's at in Shopify's codebase.
Shopify still uses DJ excessively, our version is only slightly different.
Regards
-- tobi
CEO Shopify
Dean Strelau
> I haven't had a chance to check out the 100+ forks on GitHub so it
> would be great if someone could summarize the major advantages of some
> of the third party work.
I've started a list of forks and features:
http://wiki.github.com/tobi/delayed_job/forks
I'll update this as I continue to sort through the forks. Hope
you don't mind me starting a page on your wiki.
If your fork is missing or if I've missed features, incorrectly
attributed features/fixes, or insufficiently explained something,
please feel free to step in and fix it.
Dean
Tiago Pinto
wiki page.
Btw, one of the changes we were planning was adding a unique id to
each job (e.g. "initial_fetch_15" for a job using the InitialFetch
class and for user ID 15) so we can avoid duplicate jobs for the same
data.
-t
On Oct 10, 2:38 pm, Dean Strelau <d...@strelau.net> wrote:
BrianTheCoder
feature to add to dj I think Tobi's idea of a better daemon that
manages processes like unicorn would probably benefit everyone the
most, plus be an awesome and unique feature to have for a
backgrounding library.
Brandon Keepers
On Oct 11, 2009, at 5:14 PM, BrianTheCoder wrote:
> Personally, i think that if we're going to focus as a group on a
> feature to add to dj I think Tobi's idea of a better daemon that
> manages processes like unicorn would probably benefit everyone the
> most, plus be an awesome and unique feature to have for a
> backgrounding library.
I agree. I'm fascinated by this idea and plan to play around with it
this week.
Brandon
woahdae
> feature to add to dj I think Tobi's idea of a better daemon that
> manages processes like unicorn would probably benefit everyone the
> most, plus be an awesome and unique feature to have for a
> backgrounding library.
monitor by mashing together Spawn, DaemonSpawn and DelayedJob (http://
github.com/woahdae/delayed_job_spawner). It's a simplest-thing-that-
works approach, but I had two thoughts after putting it together:
1) My only complaint about DelayedJob is that the Job class is
responsible for both scheduling and running. I'm not super familiar
with messaging design patterns, but it at least seems best to have one
thing responsible for scheduling, locking, etc, and let Job handle
deserialization and running the task. Maybe the job would out of
necessity need to handle locking for the sake of reliability, but
either way if it was done this way you could have one daemon run as
the Worker/Scheduler, and the jobs run as a Rack application. I have a
friend who made a queue runner (http://github.com/dbalatero/
queue_stick) out of sinatra, and used amazon S3 as the scheduler.
Haven't seen or used the code, but the concept is great. Letting
Passenger or Unicorn run the jobs, you obviously get all the memory-
saving and stability goodness of the projects, etc. I think this type
of mash-up would be ideal.
2) If I had to roll my own, it'd be great to have a generic Ruby
library that handles what I see as an emerging pattern:
* Preload code (with hook for custom preloading)
* Fork a bunch (w/ after_fork hook as well)
* Monitor doing stuff
Passenger, Unicorn, Spork (kinda), and my DelayedJobSpawner all follow
this pattern. My spawner is really just a ~100-line script for this,
but you could write a pretty lightweight gem to abstract this out. In
the near future, even, I need to follow this pattern for parallel-
izing migrations at work.
Anyways, if you could get this in a Rack app that seems ideal. if
you're looking to roll your own, consider this a +1 for making a gem
with a fork-anything interface. If nothing else, then,
DelayedJobSpawner is a proof of concept that the code relating to
DelayedJob would be extremely minimal (I think it's just a couple
lines in the project).
Woody Peterson
> queue_stick) out of sinatra, and used amazon S3 as the scheduler.
zilkey
of scaling issues discussed above, however we use guids as ids, so my
fork just adds support for that.
Once you make the announcement about the canonical repo, I'd be happy
to commit my changes. Thanks for taking the initiative!
Brandon Keepers
On Oct 9, 1:43 pm, Tobi <tobias.lue...@gmail.com> wrote:
> The reason is that rails cannot easily create indexes that accelerate
> "DJ's aquire new jobs" query. This is partly because of inverted sort
> orders. The standard query looks for jobs order by run_at ASC,
> priority DESC. Rails has no way of creating such an index that inverts
> on the second column so you have to drop to straight SQL for it.
http://github.com/jsuchal/delayed_job/commit/0218edd07da9c685cf7be5407f4ef00f96789706
I'm not opposed to removing the priority flag, but if we can optimize
it and make it work for most cases, then I would prefer that.
Brandon
Ian Lesperance
In my case, I want to maintain a queue for each type of job as I don't
want a sudden influx of one job type to hog the queue. Instead of
picking arbitrary numbers and assigning those numbers to different
workers, I added support for assigning specific job types to workers:
http://github.com/ezpub/delayed_job/commit/6eba7750532f52f4bb9b806977bcba45bd065141
This too could probably benefit from an index, but I'm curious if my
use case is at all similar to any of yours.
Ian
Brandon Keepers
> I'm curious what people use the priorities for.
I've only used it in one application. When a new record is created, it
uses delayed_job to update the search index and then send out email
notifications. Determining who receives the email notification
depends on the search index, so I schedule the search index job with a
higher priority than the email notifications to make sure that happens
first. I could probably just schedule the email notification job for
1 minute later and use the run_at time as a form of priority, but if
the queue were to get backed up, I would really want that priority
field to make sure the important jobs get processed before the trivial
ones.
I'm also curious to see how other people are using it.
Brandon
Patrick Tulskie
many people have forked it and done nothing, fixed the same bugs as
someone else, or modified it for their own purposes. My particular
fork has additional columns to track historical data about what is
going on in the queue. My worker setup processes several thousand
jobs per day and I need to be able to track what is going on with the
workers, when they are getting stuck, what sort of errors are most
prevalent, etc. I switched this project over from Starling/Workling
specifically for this ability.
I do agree with Tobi though - it does feel like a ticking time bomb.
My table has the proper indexes and lookups are generally quick, but I
still feel like it needs to be greatly refined in how things work. I
do use priorities in order to get faster jobs that I want done NOW to
go before a job that might take over a minute to complete, so I'm not
sure how I feel about removing the feature completely. I rely on DJ
to do a lot of stuff quickly, behind the scenes, outside of the view
layer of Rails.
I've been kicking around the idea of re-writing DJ to work with JRuby
and use system level threads rather than having to spawn 100+
instances in order to do the same job, but I am not sure this would be
a solution that works for everyone. Another idea I had was a Phusion
Passenger-esqe arrangement that responds to queue size and spawns
workers when there is demand. Daemonizing a ton of workers and using
god/monit to manage them really sucks, but I feel like it's the only
way to do things right now. A Unicorn-type solution would also be
good.
So Dean - in response to your original question, I think it's a good
idea, but it's going to be really tough because lots of people have
modified DJ to satisfy their own needs. I think if we had a giant
list of who's fork did what, we could consolidate and pull the best
features an enhancements into the main branch but until we know from
each author what the branches do then you're looking at an uphill
battle. Is there a wiki or something where we could all list this
information? My particular fork is derived from Tobi and rebased with
collectiveidea and helder because I needed their stuff - if that gives
you an idea of the current state of things.
