Restarting worker

[koops]

When I use the script/delayed_job stop and restart commands on Ubuntu
9.10, nothing happens. I have to kill -9 the process and start a new
worker.

This makes it hard to do a clean restart from Capistrano. Has anyone
solved/experienced this problem?

[Brandon Keepers]

Do you have long running jobs? The process will wait for the current job to finish. There are problems with the daemon, but I haven't had it hang and never restart.

Brandon

[Irish]

I use to have that problem also, so instead of using the daemons gem I
started using Mike Hale's Daemons gem fork. It adds a fix to kill
daemon processes after a time you specify. Check it out here
http://github.com/mikehale/daemons I haven't had any problems with
daemons not dieing since.

[Michael Guterl]

I have found that this approach is unreliable too. You end up killing
a worker who isn't finished and it leaves their job locked so no one
else can pick it up.

I have not been happy with daemon functionality in any version of
delayed_job. For a time I thought ghazel's fork of daemons was
working well, but even with it we ended up with workers no dying and
doubling up.

runit is what both Shopify and ENTP (Tender) use for managing daemon
processes. Yesterday, we finally took the dive and moved to using
runit for managing workers. So far everything is good. Information
for configuring DJ for runit is sparse, but I'm going to blog some
stuff this weekend.

Best regards,
Michael Guterl

[Brandon Keepers]

Michael,

That would be great. Please post back here when you blog about it.

I agree that the current daemon is not reliable. I hope to rewrite it using the same techniques as unicorn uses.

Brandon

[Josh K]

Hi Brandon,

I am using the current Daemons way of managing workers for an app in
production and have been unhappy with it for a while now.

I am 50% of the way through writing a way to daemonize Delayed Job
with a parent/child monitoring architecture similar to Unicorn. Give
me a couple of days and I will put my work on github for you to look
through.

Thanks

Josh

On Jan 16, 3:08 pm, Brandon Keepers <bran...@opensoul.org> wrote:
> Michael,
>
> That would be great.  Please post back here when you blog about it.
>
> I agree that the current daemon is not reliable.  I hope to rewrite it using the same techniques as unicorn uses.
>
> Brandon
>
> On Jan 16, 2010, at 8:34 AM, Michael Guterl wrote:
>
>
>
> > I have found that this approach is unreliable too.  You end up killing
> > a worker who isn't finished and it leaves their job locked so no one
> > else can pick it up.
>
> > I have not been happy with daemon functionality in any version of
> > delayed_job.  For a time I thought ghazel's fork of daemons was
> > working well, but even with it we ended up with workers no dying and
> > doubling up.
>
> > runit is what both Shopify and ENTP (Tender) use for managing daemon
> > processes.  Yesterday, we finally took the dive and moved to using
> > runit for managing workers.  So far everything is good.  Information
> > for configuring DJ for runit is sparse, but I'm going to blog some
> > stuff this weekend.
>
> > Best regards,
> > Michael Guterl
>

[Michael Guterl]

On Sat, Jan 16, 2010 at 9:08 AM, Brandon Keepers <bra...@opensoul.org> wrote:
> Michael,
>
> That would be great.  Please post back here when you blog about it.
>

Here's the article, hopefully it can be of assistance to someone:
http://diminishing.org/?p=91

> I agree that the current daemon is not reliable.  I hope to rewrite it using the same techniques as unicorn uses.

<snip>

I am extremely interested in a Unicorn style backend. We've been
using Unicorn for a couple months and it has been amazing.

Best regards,
Michael Guterl

[Michael Guterl]

On Sat, Jan 16, 2010 at 8:34 AM, Michael Guterl <mgu...@gmail.com> wrote:
> I have found that this approach is unreliable too.  You end up killing
> a worker who isn't finished and it leaves their job locked so no one
> else can pick it up.
>
> I have not been happy with daemon functionality in any version of
> delayed_job.  For a time I thought ghazel's fork of daemons was
> working well, but even with it we ended up with workers no dying and
> doubling up.
>
> runit is what both Shopify and ENTP (Tender) use for managing daemon
> processes.  Yesterday, we finally took the dive and moved to using
> runit for managing workers.  So far everything is good.  Information
> for configuring DJ for runit is sparse, but I'm going to blog some
> stuff this weekend.

<snip>

I may have spoke too soon, it seems that we have the same issue with
the long running rake task too. It seems like eventually the process
stops responding to all signals (except KILL obviously). I am
completely perplexed as to why this is happening. At least when runit
is managing the process duplicates do not occur tho, so that is one
plus.

Best,
Michael Guterl