Daemonizing ruote engine fails

[Lukas Loesche]

I'm trying to use the 'daemons' gem to daemonize the ruote engine for a single workflow. However I seem to be doing something fundamentally wrong.

My code in it's simplest form:

  wfid = ruote.launch(workflow)

  puts wfid

  Daemons.daemonize

  r = ruote.wait_for(wfid, :timeout => 10800)

I get the workflow ID but then the workflow doesn't run. I used noisy output and it just doesn't output anything. I added some debug 'puts' after the Daemons.daemonize and ruote.wait_for calls and the code get's to the first but not the second output, so it hangs somewhere in ruote.wait_for.

I also tried Process.daemon in ruby 2.0 as well as a simple

    exit if fork

    Process.setsid

    exit if fork

    ... redirect outputs to /dev/null, etc. ...

But they all yield the same result. The process outputs the workflow ID, daemonizes and is running in the process list, but it is not doing anything. By that I mean that the first participant in the workflow is supposed to write some data to a redis server and even before that it puts some text to stdout which should show up in the daemon log file, but it never does.

Interestingly if I call Daemons.daemonize({ontop: true}) which will make it stay on top and not fork or detach from the terminal, than it works and the workflow executes.

I also tried daemonizing the Process before launching the workflow but that didn't work either. Meaning the workflow launches and the wfid is written to the daemon log, but after that the first participant in the workflow isn't doing anything just like before.

Any suggestions what I might be doing wrong or what I could try to better debug the problem?

[Kenneth Kalmer]

Hi Lukas!

On Thu, Sep 12, 2013 at 10:59 AM, Lukas Loesche <lloe...@gmail.com> wrote:

Interestingly if I call Daemons.daemonize({ontop: true}) which will make it stay on top and not fork or detach from the terminal, than it works and the workflow executes.

I also tried daemonizing the Process before launching the workflow but that didn't work either. Meaning the workflow launches and the wfid is written to the daemon log, but after that the first participant in the workflow isn't doing anything just like before.

Any suggestions what I might be doing wrong or what I could try to better debug the problem?

This is indeed a sticky situation... I have some suggestions that might help.

First off, always do "Thread.abort_on_exception = true" so a an uncaught exception kills your daemon, and not just the thread.

Secondly, you could use safely (https://github.com/kennethkalmer/safely/wiki/Backtrace) to try and log a backtrace for you when then daemon dies to see if there are possible exceptions.

The only thing I can think off is that your participant possibly relies on some IO object that got opened BEFORE daemonizing, and is no longer available after daemonization...

If you could package the smallest possible example in a gist it would greatly help us to debug the issue, including the contents of your Gemfile (even if you're not showing us the code used by said gems).

Kind regards

--
Kenneth Kalmer
kenneth...@gmail.com
http://opensourcery.co.za
@kennethkalmer

[Lukas Loesche]

The only thing I can think off is that your participant possibly relies on some IO object that got opened BEFORE daemonizing, and is no longer available after daemonization...

Thank you, that was it!

Just before forking I now checked all open IO objects using:

  ObjectSpace.each_object(IO) { |f| puts f unless f.closed? }

and found out that the logger class I'm using holds the handle to it's log in a class variable. Using Thread.abort_on_exception = true as you suggested I can now see the process terminate with a uninitialized stream exception.

[Kenneth Kalmer]

Prost! Glad it worked!