Delayed Job / Servolux Preforking Worker Pool

[Michael Guterl]

Anyone out there thinking about using Servolux's[1] Preforking Worker
Pool[2] to be able to prefork Delayed::Job workers? The example[3]
uses beanstalkd, but I'm wondering what is involved with forking and
reloading the Rails environment.

I'm thinking that using Ruby Enterprise Edition and it's copy-on-write
enhancements a good deal of memory could be saved.

Thoughts?

Michael Guterl

[1] http://github.com/TwP/servolux
[2] http://github.com/TwP/servolux/blob/master/lib/servolux/prefork.rb
[3] http://github.com/TwP/servolux/blob/master/examples/beanstalk.rb

[Woody Peterson]

I'm currently using http://github.com/woahdae/delayed_job_spawner ,
which is a really simple mash-up I wrote between spork, DaemonSpawn,
and Delayed Job. It's ruby-ee copy-on-write friendly. We're currently
looking to upgrade to something that can handle load better, either a
version of delayed job that has one queue worker and multiple runners
or switch to resque, but delayed_job_spawner has been serving us well
in production and is really not that complicated. Maybe you could get
some ideas for what's needed in it.

-Woody

[David Genord II]

I transformed delayed job to have a single process which pulls jobs off the queue and forks itself to handle each job. This allows delayed job to scale back to a single process when idle. I have manually tested it pretty thoroughly against 2.3 and 2.0 (requires 2-0-compatible branch) but I need to figure out how to get the actual test suite working for all the fork stuff.

http://github.com/xspond/delayed_job

David Genord II