Redirect stdout and stderr preserving the original order
Radoulov, Dimitre
There was a question on a web forum on how to redirect stdin and stderr
in the following manner:
- two separate files for both streams
- a third file containing both streams in their original order
We haven't come up with a solution and I'm curious to hear your opinion.
For instance, let's say we have the following shell script:
% cat s
#!/bin/sh
for i in 1 2; do
for j in 1 2; do
printf '%s\n' "out $i"
done
for k in 1 2; do
printf '%s\n' "err $i" >&2
done
done
The script produces the following output on the terminal (which of
course includes both stdout and stderr streams):
% ./s
out 1
out 1
err 1
err 1
out 2
out 2
err 2
err 2
Note that we use out/err n only for clarity and easy debugging. We
assume that in the real world both the content and the order of the
stdout and stderr streams are unknown.
Now the question is, is it possible to end up with the following files
after running the script:
out.log
out 1
out 1
out 2
out 2
err.log
err 1
err 1
err 2
err 2
both.log
out 1
out 1
err 1
err 1
out 2
out 2
err 2
err 2
Note the order of the entries in both.log.
We also assume that it's not possible to modify the script/program that
generates the output in any way.
Best regards
Dimitre
Seebs
> Hi all,
> There was a question on a web forum on how to redirect stdin and stderr
> in the following manner:
>
> - two separate files for both streams
> - a third file containing both streams in their original order
>
> We haven't come up with a solution and I'm curious to hear your opinion.
Hmm.
I suspect you can, but would have to write a pretty specialized program.
Here's my idea. You write a program which creates a single unix domain
socket, and allows multiple connections to it. You then open it twice --
not using 2>&1, but
>socket 2>socket
and then that program handles the interleaving, duplicating, and so on.
You might be able to fake something up by simply prepending microsecond
timestamps to both streams, logging them, then sorting those streams together
to get the "original order".
-s
--
Copyright 2009, all wrongs reversed. Peter Seebach / usenet...@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
Radoulov, Dimitre
> On 2009-12-29, Radoulov, Dimitre<cicho...@gmail.com> wrote:
>> Hi all,
>> There was a question on a web forum on how to redirect stdin and stderr
>> in the following manner:
>>
>> - two separate files for both streams
>> - a third file containing both streams in their original order
>>
>> We haven't come up with a solution and I'm curious to hear your opinion.
[...]
> I suspect you can, but would have to write a pretty specialized program.
> Here's my idea. You write a program which creates a single unix domain
> socket, and allows multiple connections to it. You then open it twice --
> not using 2>&1, but
> >socket 2>socket
> and then that program handles the interleaving, duplicating, and so on.
>
> You might be able to fake something up by simply prepending microsecond
> timestamps to both streams, logging them, then sorting those streams together
> to get the "original order".
Interesting,
I'll try it.
Thank you!
Dimitre
mop2
Just an idea:
$ cat s
#!/bin/sh
for i in 1 2; do
for j in 1 2; do
printf '%s\n' "out $i"
done
for k in 1 2; do
printf '%s\n' "err $i" >&2
done
done
$ cat test
printf '' >b
printf '' >o
printf '' >e
tail -n0 -f o >>b & To=$!
tail -n0 -f e >>b & Te=$!
./s >o 2>e
read -t2; kill $To $Te
$ . ./test
[1]- Terminated tail -n0 -f o >> b
[2]+ Terminated tail -n0 -f e >> b
$ cat o
out 1
out 1
out 2
out 2
$ cat e
err 1
err 1
err 2
err 2
$ cat b
out 1
out 1
err 1
err 1
out 2
out 2
err 2
err 2
$
Seebs
> tail -n0 -f o >>b & To=$!
Nice! My worry would be that I don't have very high confidence
that it'll respond immediately to updates in the files -- meaning
that you could end up with things getting put in unexpected orders.
There is also the question of whether output going to a file will
result in buffering behavior, especially for stdout.
Radoulov, Dimitre
[...]
[...]
Thanks, but I don't get the correct order in b:
% head -10 s test
==> s <==
#!/bin/sh
for i in 1 2; do
for j in 1 2; do
printf '%s\n' "out $i"
done
for k in 1 2; do
printf '%s\n' "err $i" >&2
done
done
==> test <==
printf '' >b
printf '' >o
printf '' >e
tail -n0 -f o >>b & To=$!
tail -n0 -f e >>b & Te=$!
./s >o 2>e
read -t2; kill $To $Te
Here is what I get:
zsh-4.3.10[t]% . ./test
[2] 1851
[3] 1852
zsh-4.3.10[t]%
[3] + terminated tail -n0 -f e >> b
zsh-4.3.10[t]%
[2] + terminated tail -n0 -f o >> b
zsh-4.3.10[t]% head -8 e o b
==> e <==
err 1
err 1
err 2
err 2
==> o <==
out 1
out 1
out 2
out 2
==> b <==
out 1
out 1
out 2
out 2
err 1
err 1
err 2
err 2
zsh-4.3.10[t]% ksh
ksh-JM 93t+ 2009-05-01$ . ./test
[1] 1857
[2] 1858
[2] + Terminated . ./test
[1] - Terminated . ./test
ksh-JM 93t+ 2009-05-01$ head -8 e o b
==> e <==
err 1
err 1
err 2
err 2
==> o <==
out 1
out 1
out 2
out 2
==> b <==
err 1
err 1
err 2
err 2
ksh-JM 93t+ 2009-05-01$
ksh-JM 93t+ 2009-05-01$ bash
bash-4.0.33(1)[t]$ . ./test
bash-4.0.33(1)[t]$ head -8 e o b
==> e <==
err 1
err 1
err 2
err 2
==> o <==
out 1
out 1
out 2
out 2
==> b <==
out 1
out 1
out 2
out 2
err 1
err 1
err 2
err 2
[1]- Terminated tail -n0 -f o >> b
[2]+ Terminated tail -n0 -f e >> b
bash-4.0.33(1)[t]$
What shell and OS are you using?
Regards
Dimitre
Icarus Sparry
> Hi all,
> There was a question on a web forum on how to redirect stdin and stderr
> in the following manner:
>
> - two separate files for both streams - a third file containing both
> streams in their original order
>
> We haven't come up with a solution and I'm curious to hear your opinion.
For typical programs, this is very hard. The reason is that almost all
programs use the "stdio" (standard I/O) library.
stdio has a default buffering strategy for stderr, that says it is
unbuffered. So sending anything to stderr makes it appear pretty much
straight away.
The strategy for stdout depends on if the library thinks it is wtiting to
a terminal or not. If it is not then the output only gets sent when an
internal buffer is full or the program is terminating. If it is writing
to a terminal then output gets sent when a line is completed.
So this means that a program
int main(int ac, char *av[]){ printf("Hello\n");
fprintf(stderr,"World\n"); return 0;}
will output the two words in a different order depending on if it is
being run with the output going to a pipe (or file) or if it is going to
a console.
If we assume that you want the "console" output order, then you need to
convince the stdio library that your program is talking to a console.
There are a couple of ways you can do this, I like an old program "pty",
but for most people "expect" is easier to find. It comes with an example
script "unbuffer" that does exactly what we want.
At this point you then just need to have a program that listens on
multiple inputs and writes to multiple outputs, and a way to tie them
together. Such a program is "multitee" (from the multitee package in
Debian).
One way to tie them together is with a named pipe.
# Create a pipe for the stderr of the program.
mkfifo sepipe
{ sleep 1 # To make sure that the shell sets up a reader for sepipe
unbuffer program 2>sepipes
rm sepipe
} | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4
Shells such as "rc" allow you to tie multiple steams together in a
pipeline in a reasonably easy way.
You might look at the thread
http://groups.google.com/group/comp.unix.shell/browse_thread/thread/
dbe5abeac7635d81/5ed189b9dfb75699
mop2
~
My system is a slackware:
$ uname -a
Linux k7 2.6.31.6 #1 PREEMPT Sat Nov 21 05:39:49 hrV 2009 i686 GNU/Linux
$ tail --version|head -n1
tail (GNU coreutils) 8.2
$ $0 --version|head -n1
GNU bash, version 4.0.35(1)-release (i686-pc-linux-gnu)
$ ls -l /lib/libc.so*
lrwxrwxrwx 1 root root 11 2009-11-20 19:47 /lib/libc.so.6 -> libc-2.9.so*
The more important here, I think, is the program tail.
If the "-s" is available you can try it to see if behavior changes for small values:
-s, --sleep-interval=N
with -f, sleep for approximately N seconds (default 1.0) between iterations
I never tried this option, but in the past I had problems with buffering, as said by others.
I really like use exclusively shell when this is not inconvenient , and this option always
solves to me buffering and delay problems.
So, a new test, using 2 shell loops in background an your script "s":
$ cat test1
printf '' >b
printf '' >o
mkfifo fo
printf '' >e
mkfifo fe
while read;do printf "$REPLY\n";printf "$REPLY\n">>b;done >o <fo &
while read;do printf "$REPLY\n";printf "$REPLY\n">>b;done >e <fe &
./s >fo 2>fe
rm fo fe
head o e b
$ . ./test1
[1]- Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >> b;
done > o < fo
[2]+ Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >> b;
done > e < fe
==> o <==
out 1
out 1
out 2
out 2
==> e <==
err 1
err 1
err 2
err 2
==> b <==
out 1
out 1
Han Pingtian
sh-4.0$ cat t.
t.sh t.txt
sh-4.0$ cat t.sh
#!/bin/bash
for i in 1 2; do
for j in 1 2; do
printf '%s\n' "out $i"
done
for k in 1 2; do
printf '%s\n' "err $i" >&2
done
done
sh-4.0$ ./t.sh 2>&1 >/dev/null
err 1
err 1
err 2
err 2
sh-4.0$ unbuffer !!
unbuffer ./t.sh 2>&1 >/dev/null
sh-4.0$
--
Han Pingtian
Radoulov, Dimitre
I tested the script on Ubuntu 9.10 on Sun VirtualBox (Windows XP guest)
and on a real (physical) RHEL 5.1.
> $ tail --version|head -n1
> tail (GNU coreutils) 8.2
>
> $ $0 --version|head -n1
> GNU bash, version 4.0.35(1)-release (i686-pc-linux-gnu)
>
> $ ls -l /lib/libc.so*
> lrwxrwxrwx 1 root root 11 2009-11-20 19:47 /lib/libc.so.6 -> libc-2.9.so*
Just for completeness:
Ubuntu VM:
$ tail --version|head -n1
tail (GNU coreutils) 7.4
$ $0 --version|head -n1
GNU bash, version 4.0.33(1)-release (i486-pc-linux-gnu)
$ ls -l /lib/libc.so*
lrwxrwxrwx 1 root root 14 2009-12-26 11:20 /lib/libc.so.6 -> libc-2.10.1.so
RHEL:
$ tail --version|head -n1
tail (GNU coreutils) 5.97
$ $0 --version|head -n1
GNU bash, version 3.1.17(1)-release (i686-redhat-linux-gnu)
$ ls -l /lib/libc.so*
lrwxrwxrwx 1 root root 11 Jan 7 2008 /lib/libc.so.6 -> libc-2.5.so
> The more important here, I think, is the program tail.
> If the "-s" is available you can try it to see if behavior changes for
> small values:
> -s, --sleep-interval=N
> with -f, sleep for approximately N seconds (default 1.0) between iterations
>
> I never tried this option, but in the past I had problems with
> buffering, as said by others.
> I really like use exclusively shell when this is not inconvenient , and
> this option always
> solves to me buffering and delay problems.
I tried with values from 0.1 to 0.5, but I get the same result.
Later I will try the test again on a different system.
The order in b is not constant between executions. This is on RHEL 5.1:
$ . ./test1
[1]- Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >>b;
done >o <fo
[2]+ Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >>b;
done >e <fe
==> o <==
out 1
out 1
out 2
out 2
==> e <==
err 1
err 1
err 2
err 2
==> b <==
out 1
out 1
out 2
out 2
err 1
err 1
err 2
err 2
$ . ./test1
[1]- Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >>b;
done >o <fo
[2]+ Done while read; do
printf "$REPLY\n"; printf "$REPLY\n" >>b;
done >e <fe
==> o <==
out 1
out 1
out 2
out 2
==> e <==
err 1
err 1
err 2
err 2
==> b <==
out 1
err 1
out 1
err 1
err 2
err 2
out 2
out 2
Thanks again for the ideas anyway.
Regards
Dimitre
Radoulov, Dimitre
Hi Icarus Sparry,
thank you very much for replying!
As another poster already mentioned, the unbuffer expect script found on
Internet combines the stdout and stderr (I don't know if that script
could be rewritten in order to handle the streams differently, I'll
check that).
> At this point you then just need to have a program that listens on
> multiple inputs and writes to multiple outputs, and a way to tie them
> together. Such a program is "multitee" (from the multitee package in
> Debian).
I didn't know about multitee, thank you!
I'll try it out.
> One way to tie them together is with a named pipe.
>
> # Create a pipe for the stderr of the program.
> mkfifo sepipe
> { sleep 1 # To make sure that the shell sets up a reader for sepipe
> unbuffer program 2>sepipes
> rm sepipe
> } | multitee 3<sepipe>stdout 2>stderr 4>both 0:1,4 3:2,4
This one appears to hang, am I missing something?
zsh-4.3.10[t]% cat s
#!/bin/sh
for i in 1 2; do
for j in 1 2; do
printf '%s\n' "out $i"
done
for k in 1 2; do
printf '%s\n' "err $i" >&2
done
done
zsh-4.3.10[t]% cat unbuffer
#!/bin/sh
#
# Description: unbuffer stdout/stderr of a program
# Author: Don Libes, NIST
#
# Option: -p allow run program to read from stdin (for simplification)
#
# Note that expect can 'continue' a comment line, so the follow re-runs this
# as a expect command regardless of its PATH location.
# \
exec expect -- "$0" ${1+"$@"}
if {[string compare [lindex $argv 0] "-p"] == 0} {
# pipeline
set stty_init "-echo"
eval spawn -noecho [lrange $argv 1 end]
interact
} else {
set stty_init "-opost"
eval spawn -noecho $argv
set timeout -1
expect
}
zsh-4.3.10[t]% cat test3
set -xv
mkfifo sepipe
{ sleep 1 # To make sure that the shell sets up a reader for sepipe
unbuffer ./s 2>sepipes
rm sepipe
} | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4
set +xv
zsh-4.3.10[t]% . ./test3
mkfifo sepipe
+./test3:3> mkfifo sepipe
{ sleep 1 # To make sure that the shell sets up a reader for sepipe
unbuffer ./s 2>sepipes
rm sepipe
} | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4
+./test3:4> sleep 1
+./test3:6> rm sepipe
^C
set +xv
+./test3:9> set +xv
> Shells such as "rc" allow you to tie multiple steams together in a
> pipeline in a reasonably easy way.
Do you mean something like the Z Shell multios option?
I didn't manage to get any better results as far as this test is
concerned (hence my post here).
> You might look at the thread
> http://groups.google.com/group/comp.unix.shell/browse_thread/thread/
> dbe5abeac7635d81/5ed189b9dfb75699
Reading it now,
thanks again!
Regards
Dimitre
Radoulov, Dimitre
> On 30/12/2009 0.27, Icarus Sparry wrote:
>
> Hi Icarus Sparry,
> thank you very much for replying!
>
>> On Tue, 29 Dec 2009 18:11:01 +0100, Radoulov, Dimitre wrote:
[...]
>> One way to tie them together is with a named pipe.
>>
>> # Create a pipe for the stderr of the program.
>> mkfifo sepipe
>> { sleep 1 # To make sure that the shell sets up a reader for sepipe
>> unbuffer program 2>sepipes
>> rm sepipe
>> } | multitee 3<sepipe>stdout 2>stderr 4>both 0:1,4 3:2,4
>
> This one appears to hung, am I missing something?
[...]
> zsh-4.3.10[t]% cat test3
> set -xv
>
> mkfifo sepipe
> { sleep 1 # To make sure that the shell sets up a reader for sepipe
> unbuffer ./s 2>sepipes
> rm sepipe
> } | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4
>
> set +xv
>
>
> zsh-4.3.10[t]% . ./test3
>
> mkfifo sepipe
> +./test3:3> mkfifo sepipe
> { sleep 1 # To make sure that the shell sets up a reader for sepipe
> unbuffer ./s 2>sepipes
> rm sepipe
> } | multitee 3<sepipe >stdout 2>stderr 4>both 0:1,4 3:2,4
> +./test3:4> sleep 1
> +./test3:6> rm sepipe
> ^C
>
> set +xv
> +./test3:9> set +xv
[...]
Just to add that the version of multitee used in the above script is 3.0-4.
Regards
Dimitre
Radoulov, Dimitre
[...]
>> On Tue, 29 Dec 2009 18:11:01 +0100, Radoulov, Dimitre wrote:
[...]
>> One way to tie them together is with a named pipe.
>>
>> # Create a pipe for the stderr of the program.
>> mkfifo sepipe
>> { sleep 1 # To make sure that the shell sets up a reader for sepipe
>> unbuffer program 2>sepipes
>> rm sepipe
>> } | multitee 3<sepipe>stdout 2>stderr 4>both 0:1,4 3:2,4
>
> This one appears to hang, am I missing something?
[...]
OK,
I was missing something, the name of the pipe was misspelled (sepipe <>
sepipes).
Anyway, as I already said, this method doesn't appear to be working with
the unbuffer expect script I'm using because of the before mentioned
stderr/stdout streams handling.
Regards
Dimitre
John Koy
[...]
>>>
>>> # Create a pipe for the stderr of the program.
>>> mkfifo sepipe
>>> { sleep 1 # To make sure that the shell sets up a reader for sepipe
>>> unbuffer program 2>sepipes
>>> rm sepipe
>>> } | multitee 3<sepipe>stdout 2>stderr 4>both 0:1,4 3:2,4
>>
>> This one appears to hang, am I missing something?
>
> [...]
>
> OK,
> I was missing something, the name of the pipe was misspelled (sepipe <>
> sepipes).
>
> Anyway, as I already said, this method doesn't appear to be working with
> the unbuffer expect script I'm using because of the before mentioned
> stderr/stdout streams handling.
>
All these tricks are useless. One cannot achieve synchronousness across
process boundaries without explicit IPC synchronization, like using
semaphores.
The example above creates 2 pipes between 2 processes, and *assumes*
that when producer writes into 1st pipe at time T and to 2nd pipe at
T+1, the consumer will read from the 1st at, say, T+2 and the 2nd at
T+3. This is wrong. There's no such guaranteed synchronous delivery.
Reading from the 1st could very well be at T+3, and the other at T+2.
Icarus Sparry
Interesting. I will have to look at unbuffer (and expect in general) a
bit more.
Thanks.
mop2
> On 30/12/2009 2.44, mop2 wrote:
>> The more important here, I think, is the program tail.
>> If the "-s" is available you can try it to see if behavior changes for
>> small values:
>> -s, --sleep-interval=N
>> with -f, sleep for approximately N seconds (default 1.0) between iterations
>>
>> I never tried this option, but in the past I had problems with
>> buffering, as said by others.
>
> I tried with values from 0.1 to 0.5, but I get the same result.
> Later I will try the test again on a different system.
When I saw this option, my first idea was the value zero or times very very
small, as .00001 sec or lower.
>> while read;do printf "$REPLY\n";printf "$REPLY\n">>b;done >o <fo &
>> while read;do printf "$REPLY\n";printf "$REPLY\n">>b;done >e <fe &
In the above proposal, 'printf "$REPLY\n">>b' before
'printf "$REPLY\n"', seems more logical.
bsh
"Dimitre Radoulov" <cicho...@gmail.com> wrote:
> ... how to redirect stdin and stderr into two separate files
> for both streams, with a third file containing both streams
> in their original order
To save stdout, stderr, and the original stream into three
separate files, try this:
# untested for your application
( ( ./myprogram 2>&1 1>&3 | tee ~/stderr.txt
) 3>&1 1>&2 |
tee ~/stdout.txt
) >~/stdmix.txt 2>&1
Icarus Sparry wrote:
> ... For typical programs, this is very hard. The reason is that
> almost all programs use the "stdio" (standard I/O) library.
This is why ksh93 utilizes the Kiem-Phong Vo's stdio emulation
library, sfio ("Safe and Fast I/O"). It is a component in AT&T's
AST library, which happens to be statically linked into ksh93,
which is why if the above doesn't work, to try to use this shell
instead. Pre-compiled version exist specifically for RHEL and many
other Unices.
John Koy <John....@example.com> wrote:
> All these tricks are useless. One cannot achieve synchronousness
> across process boundaries without explicit IPC synchronization,
> like using semaphores.
Yes and no. Since IRIX 3+, all the stdio functions and macroes
are written to preserve line-by-line serialization (in the old
sense of the term). Presumably other systems are written this
way, but I have no information. Like I indicated above, there
is a reason that sfio was created!
If indeed semaphores are your only resort, after ten years of
research and experimentation, I have come to the conclusion,
that at least until ksh has the builtin ability, the only decent
semaphore library and batch job frontend, are:
"semaphore.ksh"
http://www.unixreview.com/documents/s=9303/sam0408f/0408f.htm
http://www.samag.com/code/
ftp://ftp.mfi.com/pub/sysadmin/2004/aug2004.zip
TRACT: Schaefer, Ed and John Spurgeon. "Queuing Jobs with qjob".
2005-07. Sys Admin. 14(8). <Url:http://www.theillien.com/Sys_Admin_v12/
html/v14/i08/a8.htm>.
mop2 wrote:
> > tail -s
> When I saw this option, my first idea was the value zero
> or times very very small, as .00001 sec or lower.
Time granularity and accuracy is (very) system dependent --
if it even supports this level at all. A better idea, if
only more portable, is to try Daniel Bernstein's "tai64n":
"tai64n.c": put a precise timestamp on each line
http://cr.yp.to/daemontools/tai64n.html
In general, I _would_ have thought to have suggested looking
through his pipe-tools and/or conn-tools library, but that
now I see that its component software tools are either
deprecated or inapplicable to your use: try the above ideas
first, but if all else fails, look at:
http://www.skarnet.org/software/pipe-tools/
http://www.skarnet.org/software/conn-tools/
=Brian
Stephane CHAZELAS
[...]
> Icarus Sparry wrote:
>> ... For typical programs, this is very hard. The reason is that
>> almost all programs use the "stdio" (standard I/O) library.
>
> This is why ksh93 utilizes the Kiem-Phong Vo's stdio emulation
> library, sfio ("Safe and Fast I/O"). It is a component in AT&T's
> AST library, which happens to be statically linked into ksh93,
> which is why if the above doesn't work, to try to use this shell
> instead. Pre-compiled version exist specifically for RHEL and many
> other Unices.
In which way does that solve the problem. Are you saying that
commands run by those shells will use a different stdio by some
sort of LD_PRELOAD trick?
> John Koy <John....@example.com> wrote:
>> All these tricks are useless. One cannot achieve synchronousness
>> across process boundaries without explicit IPC synchronization,
>> like using semaphores.
>
> Yes and no. Since IRIX 3+, all the stdio functions and macroes
> are written to preserve line-by-line serialization (in the old
> sense of the term). Presumably other systems are written this
> way, but I have no information. Like I indicated above, there
> is a reason that sfio was created!
That has nothing to do with stdio here. With pipes, you'd need
some sort of syncronisation.
The problem is that you need to have a process read from 2
pipes, and you can't guarantee that this process will read the
data in the order they were written. You could only guarantee
that if writing to the pipe was blocking until the reading
process has read everything (and thanksfully, pipes don't work
like that) and if there aren't processes writing simultaneously
to stdout and stderr.
For instance, in
echo x
echo y >&2
echo z
which results in
write(1, "x\n", 2);
write(2, "y\n", 2);
write(1, "z\n", 2);
(echo doesn't use stdio, or fflushes afterwards if it does
anyway).
What is likely to happen is that that command will perform the 3
writes at once before any other process is scheduled.
Then your process that reads the other end of those 2 pipes will
read "x\nz\n" from the first one and "y\n" from the second in no
predictable order, and if it's 2 separate processes reading the
2 pipes, the result is even less likely to be predictable.
> If indeed semaphores are your only resort, after ten years of
> research and experimentation, I have come to the conclusion,
> that at least until ksh has the builtin ability, the only decent
> semaphore library and batch job frontend, are:
>
> "semaphore.ksh"
Here you need semaphores in between write(2) and read(2) system
calls, so I don't expect a shell solution to be of any use.
a LD_PRELOAD solution with a modified stdio that would
do the logging could work assuming all the commands called by
the shell use a dynamically linked stdio..
--
Stï¿œphane
Radoulov, Dimitre
Thank you all for your valuable input.
Best regards
Dimitre
John Koy
>
> "Dimitre Radoulov"<cicho...@gmail.com> wrote:
>> ... how to redirect stdin and stderr into two separate files
>> for both streams, with a third file containing both streams
>> in their original order
>
> To save stdout, stderr, and the original stream into three
> separate files, try this:
>
> # untested for your application
> ( ( ./myprogram 2>&1 1>&3 | tee ~/stderr.txt
> ) 3>&1 1>&2 |
> tee ~/stdout.txt
> )>~/stdmix.txt 2>&1
>
No need for such redirection monkeying. Here's the same crap in Bash
(per OP's file naming)
---------------
:> both.log
./s > >(tee out.log >>both.log) 2> >(tee err.log >>both.log)
---------------
None guarantees synchronous output in the combined output file.
>
> John Koy<John....@example.com> wrote:
>> All these tricks are useless. One cannot achieve synchronousness
>> across process boundaries without explicit IPC synchronization,
>> like using semaphores.
>
> Yes and no. Since IRIX 3+, all the stdio functions and macroes
> are written to preserve line-by-line serialization (in the old
> sense of the term). Presumably other systems are written this
> way, but I have no information. Like I indicated above, there
> is a reason that sfio was created!
This has nothing to do with serialization[sic] of user-mode data structures.
A pipe is a kernel data structure. At any time the kernel could be
handling 100s of pipes between dozens of processes. No user-mode library
can control the order and time these pipes are served, unless the kernel
has some real-time extensions.
These are all beyond the scope of shell programming.
bsh
> bsh wrote:
> > "Dimitre Radoulov"<cichomit...@gmail.com> wrote:
> > > ...
> This has nothing to do with serialization[sic] of user-mode data structures.
> A pipe is a kernel data structure. At any time the kernel could be
> handling 100s of pipes between dozens of processes. No user-mode library
> can control the order and time these pipes are served, unless the kernel
> has some real-time extensions.
Like I said: yes and no. In (at least) IRIX, which I can presume
doesn't apply here, stdio functions and macros _are_ embedded in
line-oriented locking, which to my recollection is such a user-level
library that is serialized(*) on a system-level, not a process per
process level. If I remember correctly, this makes it true that all
processes' read(2)s and write(2)s are atomic -- as indeed I understand
these system calls are in BSD-derived Unices, if not (older?) Linux
kernels. Properly implemented I/O libraries (that is, sfio in lieu of
stdio) can (or will?) then (potentially?) be formally atomic.
(*) Java etal has subsumed the notion of serialization to mean data
persistence. I intend the old meaning of locking, as per use of
SysV message queues, synchronization, shared memory, RPC,
or whatever other IPC facility.
> These are all beyond the scope of shell programming.
I completely agree! and this was my thesis: ksh93 with its vastly
improved statically linked I/O -- and maybe it is also necessary that
the underlying Unix implement atomic I/O as well... -- supports my
suggestion to at least give that shell a chance; it was the potential
solution of least effort.
=Brian
bsh
> bsh wrote:
> > Icarus Sparry wrote:
> > This is why ksh93 utilizes the Kiem-Phong Vo's stdio emulation
> > library, sfio ("Safe and Fast I/O").
> In which way does that solve the problem. Are you saying that
> commands run by those shells will use a different stdio by some
> sort of LD_PRELOAD trick?
Uh, shells, plural?
Does the LD_PRELOAD facility (trick?) apply at all to _statically_
linked I/O?
I don't have ksh93 source code in front of me; I don't know if ksh93
I/O calls are interpositioned with their stdio.h equivalents or if
they
have been given a discreet namespace.
> That has nothing to do with stdio here. With pipes, you'd need
> some sort of syncronisation.
I don't and do agree. See my latest response to John Koy.
> Then your process that reads the other end of those 2 pipes will
> read "x\nz\n" from the first one and "y\n" from the second in no
> predictable order, and if it's 2 separate processes reading the
> 2 pipes, the result is even less likely to be predictable.
I do not have any actual test case that I can give a counterexample
to this (on a system that implements atomic I/O?). My systems
programmer intuition says to me that the true context of the matter
somehow subsumes both of our suppositions.
> a LD_PRELOAD solution with a modified stdio that would
> do the logging could work assuming all the commands called by
> the shell use a dynamically linked stdio.
Hmmm. Isn't "all" (well, when it isn't using sfio...) of process I/O
implemented via dynamically linked in stdio calls?
=Brian
yawtd
{
{
cmd [--line-buffered] 3>&- |
tee stdout.log 2>&3 3>&-
} 2>&1 >&4 4>&- |
tee stderr.log 2>&3 3>&-
} 3>&2 4>&1 2>&1 | tee /dev/stderr > all.log
Inspired by: http://wiki.bash-hackers.org/howto/redirection_tutorial