Modify stderr without disturbing stdout

[Kenny McCormack]

This is presented as a shell programming problem. I am reasonably sure I
could do it via a (trivial) "C helper" program, but am wondering if it is
possible to do it in pure shell.

Suppose I have a (closed source, unmodifiable) program called "cmd" that
generate normal output on stdout, that I want to pipe into another program.
So, I have:

$ cmd | filterprogram

But, cmd can also generate error messages on stderr (and set its return
value to something non-zero). What I want to do is to "edit" the messages
coming out on stderr. Say, if the message is "unknown floobie on garbo",
which is meaningless, I want instead to display "Your flotznick is borken".

I am pretty sure (I say this because I haven't done it yet) that I could
write a C program that:
1) Forks
2) Sets up a pipe to read the sub-process's stderr
3) Execs the new program
4) Reads from the pipe, looking for key strings
5) Changes those strings as described above
6) Displays (on the parent processes's stderr) the strings (some
modified, the rest verbatim).

Then the command line would look like:

$ wrapper cmd | filterprogram

But is there a way to do it in pure shell, without the need to write a new
C program?

Note: A solution is some non-C language, e.g., Perl, might be OK, but would
not be my first choice.

--
Nov 4, 2008 - the day when everything went
from being Clinton's fault to being Obama's fault.

[Bit Twister]

On Mon, 4 Jun 2018 10:45:32 +0000 (UTC), Kenny McCormack wrote:
>
> Suppose I have a (closed source, unmodifiable) program called "cmd" that
> generate normal output on stdout, that I want to pipe into another program.

While waiting for an answer you might put
bash stdout stderr pipe
in the first box at https://encrypted.google.com/advanced_search
where you might something like

About 155,000 results (0.35 seconds)

and just to get a feel about what can be done pick this selection
https://stackoverflow.com/questions/2342826/how-to-pipe-stderr-and-not-stdout
and read the whole post.

[Kenny McCormack]

In article <slrnpha941.j...@wb.home.test>,

Not interested in non-Usenet responses.

--
He must be a Muslim. He's got three wives and he doesn't drink.

[Bit Twister]

On Mon, 4 Jun 2018 13:58:41 +0000 (UTC), Kenny McCormack wrote:
>>
>>While waiting for an answer you might put
>>bash stdout stderr pipe
>>in the first box at https://encrypted.google.com/advanced_search
>>where you might something like
>>
>>About 155,000 results (0.35 seconds)
>>
>>and just to get a feel about what can be done pick this selection
>>https://stackoverflow.com/questions/2342826/how-to-pipe-stderr-and-not-stdout
>>and read the whole post.
>
> Not interested in non-Usenet responses.

Well, that is quite an asinine attitude. If you were to read
http://www.catb.org/~esr/faqs/smart-questions.html
it suggests you need to search the web before asking the question.

I for one have no desire to summarize the information provided in the
link due to your highnendasses laziness.

The misspell of highnesses was on purpose.
You have a nice day.

[Christian Weisgerber]

On 2018-06-04, Kenny McCormack <gaz...@shell.xmission.com> wrote:

> Suppose I have a (closed source, unmodifiable) program called "cmd" that
> generate normal output on stdout, that I want to pipe into another program.
> So, I have:
>
> $ cmd | filterprogram
>
> But, cmd can also generate error messages on stderr (and set its return
> value to something non-zero). What I want to do is to "edit" the messages
> coming out on stderr.

$ (cmd 2>&1 1>&3 | editprogram 1>&2) 3>&1 | filterprogram

--
Christian "naddy" Weisgerber na...@mips.inka.de

[Kenny McCormack]

In article <slrnphafhp...@lorvorc.mips.inka.de>,

Yeah, I thought it was something like that. Thanks for posting.

Ugly, but, yeah, I see where it is going.

--
I love the poorly educated.

[Barry Margolin]

In article <pf358c$1ob$1...@news.xmission.com>,

Use process substitution to pipe stderr to the editing program

cmd 2> >(editprogram >&2) | filterprogram

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

[Kenny McCormack]

In article <barmar-0EEBBF....@reader.eternal-september.org>,
Barry Margolin <bar...@alum.mit.edu> wrote:
...

>> Then the command line would look like:
>>
>> $ wrapper cmd | filterprogram
>>
>> But is there a way to do it in pure shell, without the need to write a new
>> C program?
>>
>> Note: A solution is some non-C language, e.g., Perl, might be OK, but would
>> not be my first choice.
>
>Use process substitution to pipe stderr to the editing program
>
>cmd 2> >(editprogram >&2) | filterprogram

I like it! Thanks.

--
Watching ConservaLoons playing with statistics and facts is like watching a
newborn play with a computer. Endlessly amusing, but totally unproductive.

[Kaz Kylheku]

Test case:

$ ((echo output ; echo error 1>&2) 2>&1 1>&3 \
| awk ' {print "stderr: " $0 }' 1>&2) 3>&1 \
| awk '{ print "stdout: " $0 }'
stdout: output
stderr: error

[Kenny McCormack]

In article <201806041...@kylheku.com>,

FWIW, this was my test case:

$ { date;ls kjhsdkfjh; } 2> >(nl -v5 >&2) | nl -v0

--
The randomly chosen signature file that would have appeared here is more than 4
lines long. As such, it violates one or more Usenet RFCs. In order to remain
in compliance with said RFCs, the actual sig can be found at the following URL:
http://user.xmission.com/~gazelle/Sigs/God

[Helmut Waitzmann]

Christian Weisgerber <na...@mips.inka.de>:

>$ (cmd 2>&1 1>&3 | editprogram 1>&2) 3>&1 | filterprogram

To free the finally unused file descriptor #3 for the "cmd" and
"editprogram" processes (file descriptor hygiene), one could add
"3>&-" to each of them:

$ (cmd 2>&1 1>&3 3>&- | editprogram 1>&2 3>&-) 3>&1 | filterprogram

[Kaz Kylheku]

Why? Aren't these in a subshell that will terminate, thereby closing
all its file descriptors?

If anything, wouldn't it be the outer shell, that needs to close the 3
that it temporarily used in order to hand the subshell a duplicate of
stdout?

[Helmut Waitzmann]

Kaz Kylheku <157-07...@kylheku.com>:

>On 2018-06-05, Helmut Waitzmann <nn.th...@xoxy.net> wrote:

>> To free the finally unused file descriptor #3 for the "cmd" and
>> "editprogram" processes (file descriptor hygiene), one could add
>> "3>&-" to each of them:
>>
>> $ (cmd 2>&1 1>&3 3>&- | editprogram 1>&2 3>&-) 3>&1 | filterprogram
>
>Why? Aren't these in a subshell that will terminate, thereby closing
>all its file descriptors?

Yes, the subshell running inside of the parentheses will terminate
as soon as "editprogram" terminates. But that wouldn't
necessarily close the opened file in the system's opened files
table, which the file descriptor #3 of the process running
"editprogram" is associated to.

For example, if "editprogram" forks a child, then terminates
itself, the child will keep write access to the pipe to
"filterprogram".

>If anything, wouldn't it be the outer shell, that needs to close the 3
>that it temporarily used in order to hand the subshell a duplicate of
>stdout?

The file descriptor #3 of the file descriptor table of the
subshell of the left hand side of the pipe to "filterprogram",
which is a duplicate of #1, won't survive the shell command line,
because file descriptors, which are opened/duplicated/closed as
part of a command invocation (other than the "exec" special shell
built-in command, when invoked without a program to exec), won't
survive that command invocation.

The problem is with the programs "cmd" and "editprogram", though:

In their file descriptor tables, each of them will have an opened
file descriptor #3, which they even don't know of. In the best
case, this is at least a waste of one file descriptor slot.

Also, with "editprogram", that might be a security issue, if
"editprogram" is supplied by some other party than the issuer of
the shell command line: "editprogram" should not be able to write
into the pipe to "filterprogram". Now, if "editprogram" were a
malign program, it could scan its file descriptor table by trying
to write to any of them, thus corrupting the data that
"filterprogram" is supposed to read.

[Kaz Kylheku]

On 2018-06-06, Helmut Waitzmann <nn.th...@xoxy.net> wrote:
> Kaz Kylheku <157-07...@kylheku.com>:
>>On 2018-06-05, Helmut Waitzmann <nn.th...@xoxy.net> wrote:
>
>>> To free the finally unused file descriptor #3 for the "cmd" and
>>> "editprogram" processes (file descriptor hygiene), one could add
>>> "3>&-" to each of them:
>>>
>>> $ (cmd 2>&1 1>&3 3>&- | editprogram 1>&2 3>&-) 3>&1 | filterprogram
>>
>>Why? Aren't these in a subshell that will terminate, thereby closing
>>all its file descriptors?
>
> Yes, the subshell running inside of the parentheses will terminate
> as soon as "editprogram" terminates. But that wouldn't
> necessarily close the opened file in the system's opened files
> table, which the file descriptor #3 of the process running
> "editprogram" is associated to.
>
> For example, if "editprogram" forks a child, then terminates
> itself, the child will keep write access to the pipe to
> "filterprogram".

I think I get it now. The concern is about "close on exec" of that
descriptor 3.

The syntax ( ... ) 3>&1 gives us a subshell in which 3 exists
as a copy of 1, temporarily, so that the subshell can arrange
certain indirections.

That subshell then forks processes for the left and right side of the
pipeline. These inherit the temporary descriptor 3.

The forked children, still running shell code, use descriptor 3 to
set up the indirections.

Then they exec their respective programs.

The programs don't refer to the temporary 3 that was only used for
reshuffling the descriptors; 3 should be closed in those programs.

Ideally, 3 would be marked "close on exec", the feature designed for
this sort of problem.

[Helmut Waitzmann]

Kaz Kylheku <157-07...@kylheku.com>:

>On 2018-06-06, Helmut Waitzmann <nn.th...@xoxy.net> wrote:
>> Kaz Kylheku <157-07...@kylheku.com>:
>>>On 2018-06-05, Helmut Waitzmann <nn.th...@xoxy.net> wrote:
>>
>>>> To free the finally unused file descriptor #3 for the "cmd" and
>>>> "editprogram" processes (file descriptor hygiene), one could add
>>>> "3>&-" to each of them:
>>>>
>>>> $ (cmd 2>&1 1>&3 3>&- | editprogram 1>&2 3>&-) 3>&1 | filterprogram
>>>
>>>Why? Aren't these in a subshell that will terminate, thereby closing
>>>all its file descriptors?
>>
>> Yes, the subshell running inside of the parentheses will terminate
>> as soon as "editprogram" terminates. But that wouldn't
>> necessarily close the opened file in the system's opened files
>> table, which the file descriptor #3 of the process running
>> "editprogram" is associated to.
>>
>> For example, if "editprogram" forks a child, then terminates
>> itself, the child will keep write access to the pipe to
>> "filterprogram".
>
>I think I get it now. The concern is about "close on exec" of that
>descriptor 3.

[correct description of the situation]

>Ideally, 3 would be marked "close on exec", the feature designed for
>this sort of problem.

Yes. AFAIK, POSIX does not specify, whether the shell will set
the close‐on‐exec flag for that file descriptor. "3>&-" will
tell the shell to explicitly close the file descriptor #3
before running the "editprogram" command.

[Kaz Kylheku]

On 2018-06-07, Helmut Waitzmann <nn.th...@xoxy.net> wrote:
> Yes. AFAIK, POSIX does not specify, whether the shell will set
> the close‐on‐exec flag for that file descriptor.

Which basically amounts to, I think, that it must not. Scripts must be
able to rely on creating descriptor 3, 4, ... using the redirection
operators such that the executed program can use these descriptors.

[Geoff Clare]

That's true for explicit redirections on commands (which is the case
being discussed in this thread), but it's worth nothing that it's not
true for redirections done in the current shell environment using "exec".
The POSIX description of "exec" says:

| If exec is specified without command or arguments, and any file
| descriptors with numbers greater than 2 are opened with associated
| redirection statements, it is unspecified whether those file
| descriptors remain open when the shell invokes another utility.

--
Geoff Clare <net...@gclare.org.uk>

[Kenny McCormack]

In article <d4aque-...@ID-313840.user.individual.net>,
Geoff Clare <net...@gclare.org.uk> wrote:
...

>That's true for explicit redirections on commands (which is the case
>being discussed in this thread), but it's worth nothing that it's not
>true for redirections done in the current shell environment using "exec".
>The POSIX description of "exec" says:
>
>| If exec is specified without command or arguments, and any file
>| descriptors with numbers greater than 2 are opened with associated
>| redirection statements, it is unspecified whether those file
>| descriptors remain open when the shell invokes another utility.

Interesting. Because just within the last week or two, I wrote a script
that depends on those FDs staying open. I would argue that if they don't,
then what's the point?

Specifically, my (bash) code is:

exec 8>somefile
gawk '...'

where the gawk script references (writes to) /dev/fd/8.

It, of course, works as expected, but are you saying that's not guaranteed
by POSIX? Is it guaranteed by BASH?

P.S. There was some reason why I didn't just do:

gawk '...' 8>somefile

but I disremember what that reason was...

--
If it seems like I'm not responding to some moronic criticism that you've
posted in response to one of my posts, be aware that I do not debate with idiots.

It doesn't mean you've won anything...

[Kaz Kylheku]

On 2018-06-08, Kenny McCormack <gaz...@shell.xmission.com> wrote:
> In article <d4aque-...@ID-313840.user.individual.net>,
> Geoff Clare <net...@gclare.org.uk> wrote:
> ...
>>That's true for explicit redirections on commands (which is the case
>>being discussed in this thread), but it's worth nothing that it's not
>>true for redirections done in the current shell environment using "exec".
>>The POSIX description of "exec" says:
>>
>>| If exec is specified without command or arguments, and any file
>>| descriptors with numbers greater than 2 are opened with associated
>>| redirection statements, it is unspecified whether those file
>>| descriptors remain open when the shell invokes another utility.
>
> Interesting. Because just within the last week or two, I wrote a script
> that depends on those FDs staying open.

The bash man page doesn't mention at all that the redirections survive
into the next command. The Info manual says: "If no command is
specified, redirections may be used to affect the current shell
environment" which seems to describes what is being relied on.

> I would argue that if they don't,
> then what's the point?

If exec with redirections only doesn't create persistent redirections,
then it's a useless synonym for the colon command, which also performs
redirections that die with it.

> P.S. There was some reason why I didn't just do:
>
> gawk '...' 8>somefile

Possibly you wanted that 8 descriptor accessible in more than just
that gawk command?

> but I disremember what that reason was...

Is that a Dubyaism? Though I wouldn't misunderestimate your ability to
coin such a thing.

[Helmut Waitzmann]

Kaz Kylheku <157-07...@kylheku.com>:

>On 2018-06-08, Kenny McCormack <gaz...@shell.xmission.com> wrote:
>> In article <d4aque-...@ID-313840.user.individual.net>,
>> Geoff Clare <net...@gclare.org.uk> wrote:
>> ...
>>>That's true for explicit redirections on commands (which is the case
>>>being discussed in this thread), but it's worth nothing that it's not
>>>true for redirections done in the current shell environment using "exec".
>>>The POSIX description of "exec" says:
>>>
>>>| If exec is specified without command or arguments, and any file
>>>| descriptors with numbers greater than 2 are opened with associated
>>>| redirection statements, it is unspecified whether those file
>>>| descriptors remain open when the shell invokes another utility.
>>
>> Interesting. Because just within the last week or two, I wrote a script
>> that depends on those FDs staying open.
>
>The bash man page doesn't mention at all that the redirections survive
>into the next command. The Info manual says: "If no command is
>specified, redirections may be used to affect the current shell
>environment" which seems to describes what is being relied on.
>
>> I would argue that if they don't,
>> then what's the point?
>
>If exec with redirections only doesn't create persistent redirections,
>then it's a useless synonym for the colon command, which also performs
>redirections that die with it.

I dont' think so. Compare

( : 8>somefile && gawk '...' 8>&8 )

with

( exec 8>somefile && gawk '...' 8>&8 )

and compare

( : 8>somefile &&
for c in a b c
do
printf '%s\n' "$c"
done >&8
)

with

( exec 8>somefile &&
for c in a b c
do
printf '%s\n' "$c"
done >&8
)

[Geoff Clare]

Kenny McCormack wrote:

> In article <d4aque-...@ID-313840.user.individual.net>,
> Geoff Clare <net...@gclare.org.uk> wrote:
> ...
>>That's true for explicit redirections on commands (which is the case
>>being discussed in this thread), but it's worth nothing that it's not
>>true for redirections done in the current shell environment using "exec".
>>The POSIX description of "exec" says:
>>
>>| If exec is specified without command or arguments, and any file
>>| descriptors with numbers greater than 2 are opened with associated
>>| redirection statements, it is unspecified whether those file
>>| descriptors remain open when the shell invokes another utility.
>
> Interesting. Because just within the last week or two, I wrote a script
> that depends on those FDs staying open. I would argue that if they don't,
> then what's the point?

They stay open in the shell itself, but are not necessarily inherited
as open file descriptors by utilities that the shell invokes.

> Specifically, my (bash) code is:
>
> exec 8>somefile
> gawk '...'
>
> where the gawk script references (writes to) /dev/fd/8.
>
> It, of course, works as expected, but are you saying that's not guaranteed
> by POSIX?

Yes, POSIX allows the shell to execute gawk with fd 8 closed.

--
Geoff Clare <net...@gclare.org.uk>

[Kenny McCormack]

In article <6hc2ve-...@ID-313840.user.individual.net>,
Captain Obvious <net...@gclare.org.uk> wrote:
...

>They stay open in the shell itself, but are not necessarily inherited
>as open file descriptors by utilities that the shell invokes.

Yup. Dumb.

...

>Yes, POSIX allows the shell to execute gawk with fd 8 closed.

Yup. Useless.

BTW, I always thought (prior to the rise of this Usenet thread) that the
main point of the "command-less" exec was specifically so that you could
re-direct all subsequent output to a file (i.e., without having to put "> file"
on each and every command). Like this:

exec > "mylogfile.log" 2>&1

(sort of a "poor man's" script(1)).

If that's not supposed to work (per "POSIX"), then something is clearly amiss.

--
A Catholic woman tells her husband to buy Viagra.

A Jewish woman tells her husband to buy Pfizer.

[Lew Pitcher]

This /may/ be a side-effect of the lee-way that POSIX gives shell
implementations. 'exec' is one of the shell "Special Builtin Utilities", and
POSIX says of them that they "need not be provided in a manner accessible
via the exec family of functions". This implies that implementations MAY
implement these utilities (including 'exec') as external utility programs.

In such a case, 'exec' would change it's own environment, and (upon
termination) lose all those changes. The environment in the invoking shell
would not be changed.

--
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request

[Kaz Kylheku]

On 2018-06-11, Kenny McCormack <gaz...@shell.xmission.com> wrote:
> In article <6hc2ve-...@ID-313840.user.individual.net>,
> Captain Obvious <net...@gclare.org.uk> wrote:
> ...
>>They stay open in the shell itself, but are not necessarily inherited
>>as open file descriptors by utilities that the shell invokes.
>
> Yup. Dumb.

If, say, FD 13 stays open in the shell environment, but doesn't get
passed to exec-ed programs, you can always duplicate it to some other
descriptor, like 7:

utility 7<&13 # repeated in every command

so 13 gets closed across the exec, but the duplicate 7 survives.

It is gratuitously inconvenient compared to just letting 13 be used.

[Barry Margolin]

In article <pfm6cc$sjh$1...@dont-email.me>,

Lew Pitcher <lew.p...@digitalfreehold.ca> wrote:

> Geoff Clare wrote:
>
> > Kaz Kylheku wrote:
> >
> >> On 2018-06-07, Helmut Waitzmann <nn.th...@xoxy.net> wrote:
> >>> Yes. AFAIK, POSIX does not specify, whether the shell will set

> >>> the close-on-exec flag for that file descriptor.

> >>
> >> Which basically amounts to, I think, that it must not. Scripts must be
> >> able to rely on creating descriptor 3, 4, ... using the redirection
> >> operators such that the executed program can use these descriptors.
> >
> > That's true for explicit redirections on commands (which is the case
> > being discussed in this thread), but it's worth nothing that it's not
> > true for redirections done in the current shell environment using "exec".
> > The POSIX description of "exec" says:
> >
> > | If exec is specified without command or arguments, and any file
> > | descriptors with numbers greater than 2 are opened with associated
> > | redirection statements, it is unspecified whether those file
> > | descriptors remain open when the shell invokes another utility.
>
> This /may/ be a side-effect of the lee-way that POSIX gives shell
> implementations. 'exec' is one of the shell "Special Builtin Utilities", and
> POSIX says of them that they "need not be provided in a manner accessible
> via the exec family of functions". This implies that implementations MAY
> implement these utilities (including 'exec') as external utility programs.
>
> In such a case, 'exec' would change it's own environment, and (upon
> termination) lose all those changes. The environment in the invoking shell
> would not be changed.

That wouldn't be valid, because it IS required to propagate changes to
FDs 0, 1, and 2. The leeway is only for higher FDs.

[Joerg.S...@fokus.fraunhofer.de]

In article <pfe0cn$31s$1...@news.xmission.com>,

Kenny McCormack <gaz...@shell.xmission.com> wrote:

>Interesting. Because just within the last week or two, I wrote a script
>that depends on those FDs staying open. I would argue that if they don't,
>then what's the point?
>
>Specifically, my (bash) code is:
>
> exec 8>somefile
> gawk '...'
>
>where the gawk script references (writes to) /dev/fd/8.
>
>It, of course, works as expected, but are you saying that's not guaranteed
>by POSIX? Is it guaranteed by BASH?

Is is not granted because the shell nay open the file with the close on exec
flag set.

This is not true for the Bourne Shell and bash, but it is true for e.g. ksh93,
which is what bash was modelled against.

BTW: it is easy to check. Run:

ls -l /proc/$$/fd

to see the files that are open in the shell and:

ls -l /proc/self/fd

to see the files that are open in "ls".

--
EMail:jo...@schily.net (home) Jörg Schilling D-13353 Berlin
joerg.s...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/

[Joerg.S...@fokus.fraunhofer.de]

In article <6hc2ve-...@ID-313840.user.individual.net>,

e.g. by using the close on exec flag.

BTW: this seems to be a feature that could be standardized to depend on a new
shell flag that can be change via "set -o".

[Joerg.S...@fokus.fraunhofer.de]

In article <pfm6cc$sjh$1...@dont-email.me>,
Lew Pitcher <lew.p...@digitalfreehold.ca> wrote:

>This /may/ be a side-effect of the lee-way that POSIX gives shell
>implementations. 'exec' is one of the shell "Special Builtin Utilities", and
>POSIX says of them that they "need not be provided in a manner accessible
>via the exec family of functions". This implies that implementations MAY
>implement these utilities (including 'exec') as external utility programs.

POSIX just requires that you are able to call execl() on that command.

It is impossible to implement the standardized features of "exec" as an
external command.

[Geoff Clare]

Kenny McCormack wrote:

> In article <6hc2ve-...@ID-313840.user.individual.net>,
> Captain Obvious <net...@gclare.org.uk> wrote:
> ...
>>They stay open in the shell itself, but are not necessarily inherited
>>as open file descriptors by utilities that the shell invokes.
>
> Yup. Dumb.
>
> ...
>>Yes, POSIX allows the shell to execute gawk with fd 8 closed.
>
> Yup. Useless.

You can use them in later commands (e.g. the way Kaz suggested in his
reply, although fd 13 isn't portable - only 0 to 9 are required to be
supported).

> BTW, I always thought (prior to the rise of this Usenet thread) that the
> main point of the "command-less" exec was specifically so that you could
> re-direct all subsequent output to a file (i.e., without having to put "> file"
> on each and every command). Like this:
>
> exec > "mylogfile.log" 2>&1
>
> (sort of a "poor man's" script(1)).
>
> If that's not supposed to work (per "POSIX"), then something is clearly amiss.

POSIX requires that usage to work. File descriptors 0, 1 and 2 are
always inherited by invoked utilities. It is only those greater than
2 that might not be.

--
Geoff Clare <net...@gclare.org.uk>