Change to DJ forking model

[Matt Griffin]

Hello,

I've been experimenting with a change to DJ's forking model. Instead
of just forking N processes up front, we have one master worker which
forks child workers when there is work available for them and lets
them die out when there isn't. This allows a worker box to
dynamically scale up (to a configured threshold) when there's a lot of
queue traffic but also to scale down and avoid memory bloat and
unnecessary DB traffic when there isn't. Monit and other process
monitoring solutions also have an easier time with this setup since
there is just one process to start and one master pid file to monitor.

This branch is against DJ 2.0 and is compatible with Rails 2.3 only.
https://github.com/Viximo/delayed_job/commits/forking

Any feedback would be appreciated.

Cheers,
Matt

[David Genord II]

I have been working on this for a while now. The current variation can be found at https://github.com/xspond/delayed_job/tree/multi-process-rework

David Genord II

[Sung Pae]

On 13 Nov 2010, at 9:19 AM, Matt Griffin wrote:

> This branch is against DJ 2.0 and is compatible with Rails 2.3 only.
> https://github.com/Viximo/delayed_job/commits/forking

I think Brandon is dropping Rails 2.3 compatibility in the near future.

> Any feedback would be appreciated.

It's a great idea! You should have checked the newsgroup archives though :)

http://groups.google.com/group/delayed_job/browse_thread/thread/cfcc103985705322/211d379896865cfd?lnk=gst&q=nathanvda#211d379896865cfd

Several people have working implementations of a forking daemon, and Brandon has been working on one himself. I think however that everybody's been pretty busy, so there just hasn't been much activity recently.

You may be interested in comparing with my version, which seems similar to yours on quick scan:

https://github.com/guns/delayed_job/blob/delayed_job_daemon/lib/delayed/daemon_tasks.rb

It's a very simple implementation (Brandon is looking for a more sophisticated overhaul of Delayed::Worker), but it's been working well for myself and a few others for a few months now.

Cheers,
guns