Spawn child process with stdout & stderr going to the same file

[Tim Cuthbertson]

I want to spawn a child process and have *all* output go to the same file.

I tried this:

    var fs = require('fs'), childProcess = require('child_process');

    var output = fs.openSync('/tmp/output.log', 'w');

    childProcess.spawn('bash', ['-c', 'echo STOUT; echo STDERR >&2; echo DONE;'], {stdio: ['ignore', output, output]});

Which presumably provides the same FD for both stdout and stderr in the child process. Looking at the file produced, I get:

    STOUT

    DONE

stderr is apparently being ignored. Looking at the child_process docs, there's an example that opens *two* versions of the same file (in append mode), and uses this for stdout / stderr. Since I want to truncate the file (not append), I tried:

    var fs = require('fs'), childProcess = require('child_process');

    var output = fs.openSync('/tmp/output.log', 'w');

    var output2 = fs.openSync('/tmp/output.log', 'a');

    childProcess.spawn('bash', ['-c', 'echo STOUT; echo STDERR >&2; echo DONE;'], {stdio: ['ignore', output, output2]});

But it looks like the write stream is not playing nice and is overwriting the other stream's output, as I get:

    STOUT

    DONE

    R

So I guess I have to explicitly truncate the file first, then open two append streams to it?

    fs.closeSync(fs.openSync('/tmp/output.log', 'w'));

    var output = fs.openSync('/tmp/output.log', 'a');

    var output2 = fs.openSync('/tmp/output.log', 'a');

    childProcess.spawn('bash', ['-c', 'echo STOUT; echo STDERR >&2; echo DONE;'], {stdio: ['ignore', output, output2]});

Which finally gives me the desired output:

    STOUT

    STDERR

    DONE

For kicks, I also tried reusing a single append stream:

    fs.closeSync(fs.openSync('/tmp/output.log', 'w'));

    var output = fs.openSync('/tmp/output.log', 'a');

    childProcess.spawn('bash', ['-c', 'echo STOUT; echo STDERR >&2; echo DONE;'], {stdio: ['ignore', output, output]});

But that gave me the same results as my initial attempt (no stderr at all).

I guess I've discovered *how* to do this (my third attempt), but that seems pretty ugly. Can anyone suggest a better way? In python, we can explicitly redirect the child's stderr to stdout:

    output = open('/tmp/output.log', 'w');

    subprocess.Popen(['bash', '-c', 'echo STOUT; echo STDERR >&2; echo DONE;'], stderr = subprocess.STDOUT)

Is there anything similar in nodejs, or am I stuck with explicitly truncating the file, followed by opening two append-mode descriptors?

Cheers,

 - Tim.

[Robert Steckroth]

Well, not to be all "let me tell ya", but if your will..

I would not use bash as a specified shell.

1. The users login shell which owns the script should be used.

2. This needs to be written in bash and called as a script.

3. DONE is a bash namespace

Logs should be used instead.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

<surgemcgee> Systems/Software Engineer

[Tim Cuthbertson]

I probably should clarified that I'm not actually running this silly inline bash script in my real code - that was to generate some sample output. My real application is running executable scripts (python, mostly).

[Stefan Klein]

2013/10/27 Stefan Klein <st.fa...@gmail.com>

2013/10/27 Tim Cuthbertson <gfx...@gmail.com>

    childProcess.spawn('bash', ['-c', 'echo STOUT; echo STDERR >&2; echo DONE;'], {stdio: ['ignore', output, output]});

Don't you have to use "2>&1" to redirect STDERR to STDOUT?  

Sorry, haven't had coffee yet.

But still, why don't you just redirect STDERR to STDOUT when calling the child process?

[Stefan Klein]

2013/10/27 Tim Cuthbertson <gfx...@gmail.com>

    childProcess.spawn('bash', ['-c', 'echo STOUT; echo STDERR >&2; echo DONE;'], {stdio: ['ignore', output, output]});

[NiftyHat Tom Mitchell]

I think you want to look at dup() and dup2.  I see some mumble foo that

this was added to node.js back in 2010 (first patch offered that I saw).

My expectation is you would use it the same as in "C".  This link may be of value:

     http://stackoverflow.com/questions/11635219/dup2-dup-why-would-i-need-to-duplicate-a-file-descriptor

The notation "2>&1" instructs the shell to do the dup sequence of system calls

to make sure the standard file descriptors stdin, stdout, stderr and their descriptor

numbers all match expectations.   The fork() exec() file descriptor tangle is well documented

in older Unix documents.   Many modern systems assume programmers know all

this old stuff and leave it as an exercise for the student.     There is open source

simple shell like code that gets it correct... 

Anytime you have multiple writers to the same file there is a risk of confused output.

Often things are fine up to the point that system load changes and true concurrency

or time slice conflict happens.  Writes to stdout, stderr are not interlocked on many

systems.

The order can prove important:
i.e. difference between ">/dev/null 2>&1" and "2>&1 >/dev/null"

http://stackoverflow.com/questions/916900/having-trouble-with-fork-pipe-dup2-and-exec-in-c

On Saturday, October 26, 2013 6:19:50 PM UTC-7, Tim Cuthbertson wrote:

I want to spawn a child process and have *all* output go to the same file.

I tried this:

    var fs = require('fs'), childProcess = require('child_process');

    var output = fs.openSync('/tmp/output.log', 'w');

    childProcess.spawn('bash', ['-c', 'echo STOUT; echo STDERR >&2; echo DONE;'], {stdio: ['ignore', output, output]});

 

...snip ......

[Tim Cuthbertson]

That would absolutely solve my problem, but I haven't found a way to do that - is it possible?

Please note that I'm calling arbitrary processes, so I can't rely on the child to do this redirection when it starts - it must be done my by the parent process before exec()ing them.

Cheers,
 - Tim.

[Alex Kocharin]

var fs = require('fs')
  , childProcess = require('child_process')

  , stream = fs.createWriteStream('/tmp/output.log')
  , child = childProcess.spawn('bash', ['-c', 'echo STDOUT; echo STDERR >&2; echo DONE;'])

child.stderr.pipe(stream)
child.stdout.pipe(stream)

[Tim Cuthbertson]

On Monday, October 28, 2013 6:53:09 PM UTC+11, Alex Kocharin wrote:

var fs = require('fs')
  , childProcess = require('child_process')
  , stream = fs.createWriteStream('/tmp/output.log')
  , child = childProcess.spawn('bash', ['-c', 'echo STDOUT; echo STDERR >&2; echo DONE;'])

child.stderr.pipe(stream)
child.stdout.pipe(stream)

Thanks for the suggestion, although unfortunately I don't think that would work for my real scenario. I'm spawning long running tasks, which may outlive the process that spawned them (which is why I'm directing the output into a file). Presumably these streams would all stop working / die when my parent process ends.
 

[Tim Cuthbertson]

On Monday, October 28, 2013 9:50:08 AM UTC+11, NiftyHat Tom Mitchell wrote:

I think you want to look at dup() and dup2.  I see some mumble foo that

this was added to node.js back in 2010 (first patch offered that I saw).

My expectation is you would use it the same as in "C".  This link may be of value:

     http://stackoverflow.com/questions/11635219/dup2-dup-why-would-i-need-to-duplicate-a-file-descriptor

The notation "2>&1" instructs the shell to do the dup sequence of system calls

to make sure the standard file descriptors stdin, stdout, stderr and their descriptor

numbers all match expectations.   The fork() exec() file descriptor tangle is well documented

in older Unix documents.   Many modern systems assume programmers know all

this old stuff and leave it as an exercise for the student.     There is open source

simple shell like code that gets it correct... 

Anytime you have multiple writers to the same file there is a risk of confused output.

Often things are fine up to the point that system load changes and true concurrency

or time slice conflict happens.  Writes to stdout, stderr are not interlocked on many

systems.

The order can prove important:
i.e. difference between ">/dev/null 2>&1" and "2>&1 >/dev/null"

http://stackoverflow.com/questions/916900/having-trouble-with-fork-pipe-dup2-and-exec-in-c

Thanks. dup sounds precisely like what I want (and that explains why the output got garbled when I tried to simply use the same FD twice).

Sadly, I can't find any bindings for `dup` in nodejs. Most of the discussions that turn up in google seem to be a patch, followed by "you don't need that". Grepping the nodejs source itself for `dup` reveals only C code using it, no JS bindings :(
 

[Tim Oxley]

Thanks for the suggestion, although unfortunately I don't think that would work for my real scenario. I'm spawning long running tasks, which may outlive the process that spawned them (which is why I'm directing the output into a file). Presumably these streams would all stop working / die when my parent process ends.

You could just spawn it detached (check the options for spawn) but you'd need to make sure you don't spawn duplicates every time your app boots… but that sounds like copmlexity for a solved problem. If you want to ensure the streams keep living, then keep them alive the same way you would anything else, as its own standalone daemon, managed by upstart or monit or whatever. 

[Sam Roberts]

On Mon, Oct 28, 2013 at 5:07 AM, Tim Cuthbertson <gfx...@gmail.com> wrote:
> Thanks for the suggestion, although unfortunately I don't think that would
> work for my real scenario. I'm spawning long running tasks, which may
> outlive the process that spawned them (which is why I'm directing the output
> into a file). Presumably these streams would all stop working / die when my
> parent process ends.

Have you read the docs for child_process.spawn? In particular, for the
stdio array?

It looks like you want option (4), where you open a file, and provide
that file to the child proces as

[ null, f, f],

so the open file is duped in child to fd 1 and 2 (stdout and stderr).

Sam

[Tim Cuthbertson]

Good point, I missed this in my reading of those docs. It does seem very similar to what I tried with passing in the same FD twice, but the docs *do* explicitly say it dup()s the fd. Worth a shot, certainly:

    var outpath = '/tmp/output.log';
    var out = fs.createWriteStream(outpath);
    childProcess.spawn('bash', ['-c', 'echo STOUT; echo STDERR >&2; echo DONE;'], {stdio: ['ignore', out, out]});

But sadly, I get the same result as when I was passing the same FD twice:

    STOUT
    DONE

Cheers,
 - Tim.

[Oleg Verych]

Hi, Tim.

You write

I want to spawn a child process and have *all* output go to the same file.

Why? (looking down the thred...)

I'm spawning long running tasks, which may outlive the process that spawned them (which is why I'm directing the output into a file).

Fine.

It is right there in docs: "Example of detaching a long-running process and redirecting its output to a file"

Like you've found:

 

So I guess I have to explicitly truncate the file first, then open two append streams to it?

    //fs.closeSync(fs.openSync('/tmp/output.log', 'w'));

    var output = fs.openSync('/tmp/output.log', 'a');

    var output2 = fs.openSync('/tmp/output.log', 'a');

    childProcess.spawn('bash', ['-c', 'echo STOUT; echo STDERR >&2; echo DONE;'], {stdio: ['ignore', output, output2]});

Works on both MS Windows and */Linux (tested). It even looks symmetrical (by action, not by value, but anyway)

Thus, why not to just read docs and use it, instead of remembering all that legacy tricks on other languages, platforms, etc.?

BTW, this works on MS Windows without overlapping: {stdio: ['ignore', output, output]}...

_______