bash: how to pipe stdout and stderr separately, line by line

[aleks...@krediidiinfo.ee]

Hi,

I need a way to pipe stdout and stderr of a program separately to
scripts which read their input line by line. The scripts are
basically
"while read line;do logger {custom options} -- $line;done".

With stdout or stdout and stderr combined this is simple. A simple
pipe
or "2>&1 |" for both will work.

However if I want to have them separately it isn't as simple.

One solution I found is process substitution "program > >(script1.sh)
2>
>(script2.sh)", but with this script#.sh are fed all the output at
once, not line by line as I need it.

What I want would look something like "program | script1.sh 2|
script2.sh", where "2|" is a pipe for stderr.

Any ideas?

One suggestion was to use named pipes. This isn't really feasible as
I
want to use this in scripts and often.

(Originally posted at
http://forums.fedoraforum.org/showthread.php?p=1097153 , redirected
here.)

Regards,

Aleksander Kamenik
System Administrator
Krediidiinfo AS
an Experian Company
Phone: +372 665 9649
Email: aleks...@krediidiinfo.ee

http://www.krediidiinfo.ee/
http://www.experiangroup.com/

[Stephane CHAZELAS]

2008-10-16, 07:58(-07), aleks...@krediidiinfo.ee:
[...]

> I need a way to pipe stdout and stderr of a program separately to
> scripts which read their input line by line.

[...]

{ {
cmd 3>&- 4>&- | script1 >&3 2>&4 3>&- 4>&-
} 2>&1 | script2 3>&- 4>&- } 3>&1 4>&2

All the x>&-'s are for tidyness and may be ommited if you don't
care about unnecessary fds being open in those processes.

--
Stéphane

[Javi Barroso]

On Oct 16, 7:00 pm, Stephane CHAZELAS <stephane_chaze...@yahoo.fr>
wrote:
> 2008-10-16, 07:58(-07), aleksan...@krediidiinfo.ee:

Hi Stéphane,

If you could explain this code, step by step, it would be useful for
many people, and I could rate this post as "excelent" at google
groups ;)

I know ... ;-) :

RTFM
http://tldp.org/LDP/abs/html/io-redirection.html#IOREDIRREF
http://tldp.org/LDP/abs/html/ioredirintro.html

Thanks

[mop2]

On Thu, 16 Oct 2008 11:58:57 -0300, <aleks...@krediidiinfo.ee> wrote:

> I need a way to pipe stdout and stderr of a program separately to
> scripts which read their input line by line. The scripts are
> basically
> "while read line;do logger {custom options} -- $line;done".

> What I want would look something like "program | script1.sh 2|

> script2.sh", where "2|" is a pipe for stderr.
>

$ /bin/ls -l
total 8
-rwxr-xr-x 1 web ppp 41 2008-10-16 20:53 s1
-rwxr-xr-x 1 web ppp 41 2008-10-16 20:53 s2

$ cat s1
while read;do
echo $0: $REPLY >&2
done

$ cat s2
while read;do
echo $0: $REPLY >&2
done

$ { { ls -l s1 s2 s3 s4 2>&3 |./s1;} 3>&1;}|./s2
./s2: /bin/ls: cannot access s3: No such file or directory
./s2: /bin/ls: cannot access s4: No such file or directory
./s1: -rwxr-xr-x 1 web ppp 41 2008-10-16 20:53 s1
./s1: -rwxr-xr-x 1 web ppp 41 2008-10-16 20:53 s2

[Barry Margolin]

In article
<64946174-29d5-4006...@m36g2000hse.googlegroups.com>,
aleks...@krediidiinfo.ee wrote:

> Hi,
>
> I need a way to pipe stdout and stderr of a program separately to
> scripts which read their input line by line. The scripts are
> basically
> "while read line;do logger {custom options} -- $line;done".
>
> With stdout or stdout and stderr combined this is simple. A simple
> pipe
> or "2>&1 |" for both will work.
>
> However if I want to have them separately it isn't as simple.
>
> One solution I found is process substitution "program > >(script1.sh)
> 2>
> >(script2.sh)", but with this script#.sh are fed all the output at
> once, not line by line as I need it.

That's probably because your program is using stdio, which buffers
output when it's going to a pipe or file rather than a terminal. All
forms of redirection will suffer from this.

Use the "unbuffer" command that comes with Expect.

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

[aleks...@krediidiinfo.ee]

Thanks for all the replies.

I've decided to drop the idea as there's really no perfect solution.
But the insights you gave are valuable nonetheless (unbuffer,
redirects via fd# etc).

Thanks a bunch!