implicit redirection of background within pipeline

[Martin D Kealey]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu'
-DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL
-DHAVE_CONFIG_H -I. -I../. -I.././include -I.././lib -D_FORTIFY_SOURCE=2
-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat
-Werror=format-security -Wall
uname output: Linux treehug.home.kurahaupo.gen.nz 3.13.0-55-generic
#94-Ubuntu SMP Thu Jun 18 00:27:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu

Bash Version: 4.3
Patch Level: 11
Release Status: release

Description:
The first backgrounded element within a pipeline component has its
stdin redirected to /dev/null; this does not apply to the second and
subsequent background elements.

On a Linux system this can be observed with:
echo x | ( ls -ld /proc/self/fd/0 &
ls -ld /proc/self/fd/0 &
wait )

On other systems it is apparent simply with:
echo x | ( cat & wait )
# no output is produced

Repeat-By:
# this command produces no output:
echo x | ( cat & wait )

Fix:
# this is only a work-around:
echo x ( :& cat & wait )

[Piotr Grzybowski]

hey,

I am quite sure it happens here (devel branch, at
6f82653c5ef09aeeeba4376a1c65ce86c3605c00):

execute_cmd.c +5115:
if ((cmdflags & CMD_STDIN_REDIR) &&
pipe_in == NO_PIPE &&
(stdin_redirects (redirects) == 0))
async_redirect_stdin ();

but after reading the comments some 3.5k lines earlier I am not sure
how to fix it; of course commenting out the if in
execute_cmd.c:5112-5115 solves it, but who can tell what it breaks?

cheers,
pg

[Chet Ramey]

On 1/10/16 12:54 PM, Piotr Grzybowski wrote:
> hey,
>
> I am quite sure it happens here (devel branch, at
> 6f82653c5ef09aeeeba4376a1c65ce86c3605c00):
>
> execute_cmd.c +5115:
> if ((cmdflags & CMD_STDIN_REDIR) &&
> pipe_in == NO_PIPE &&
> (stdin_redirects (redirects) == 0))
> async_redirect_stdin ();
>
> but after reading the comments some 3.5k lines earlier I am not sure
> how to fix it; of course commenting out the if in
> execute_cmd.c:5112-5115 solves it, but who can tell what it breaks?

Posix requires this behavior:

"The standard input for an asynchronous list, before any explicit
redirections are performed, shall be considered to be assigned to a file
that has the same properties as /dev/null."

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU ch...@case.edu http://cnswww.cns.cwru.edu/~chet/

[Andreas Schwab]

Chet Ramey <chet....@case.edu> writes:

> On 1/10/16 12:54 PM, Piotr Grzybowski wrote:
>> hey,
>>
>> I am quite sure it happens here (devel branch, at
>> 6f82653c5ef09aeeeba4376a1c65ce86c3605c00):
>>
>> execute_cmd.c +5115:
>> if ((cmdflags & CMD_STDIN_REDIR) &&
>> pipe_in == NO_PIPE &&
>> (stdin_redirects (redirects) == 0))
>> async_redirect_stdin ();
>>
>> but after reading the comments some 3.5k lines earlier I am not sure
>> how to fix it; of course commenting out the if in
>> execute_cmd.c:5112-5115 solves it, but who can tell what it breaks?
>
> Posix requires this behavior:
>
> "The standard input for an asynchronous list, before any explicit
> redirections are performed, shall be considered to be assigned to a file
> that has the same properties as /dev/null."

But isn't it a bug that subsequent asynchronous commands in the list are
not redirected from /dev/null?

Andreas.

--
Andreas Schwab, SUSE Labs, sch...@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

[Chet Ramey]

On 1/11/16 11:37 AM, Andreas Schwab wrote:
> Chet Ramey <chet....@case.edu> writes:
>
>> On 1/10/16 12:54 PM, Piotr Grzybowski wrote:
>>> hey,
>>>
>>> I am quite sure it happens here (devel branch, at
>>> 6f82653c5ef09aeeeba4376a1c65ce86c3605c00):
>>>
>>> execute_cmd.c +5115:
>>> if ((cmdflags & CMD_STDIN_REDIR) &&
>>> pipe_in == NO_PIPE &&
>>> (stdin_redirects (redirects) == 0))
>>> async_redirect_stdin ();
>>>
>>> but after reading the comments some 3.5k lines earlier I am not sure
>>> how to fix it; of course commenting out the if in
>>> execute_cmd.c:5112-5115 solves it, but who can tell what it breaks?
>>
>> Posix requires this behavior:
>>
>> "The standard input for an asynchronous list, before any explicit
>> redirections are performed, shall be considered to be assigned to a file
>> that has the same properties as /dev/null."
>
> But isn't it a bug that subsequent asynchronous commands in the list are
> not redirected from /dev/null?

Yes, clearly the existing behavior is inconsistent. My response answered
Piotr's question about what it breaks.

[Chet Ramey]

On 1/11/16 3:29 PM, Martin Kealey wrote:
> Surely a pipe counts as an explicit redirection?

It doesn't.

> And even if it didn't, the inconsistency between the first and subsequent
> asynchronous elements is confusing.

Agreed.

[Chet Ramey]

On 1/11/16 3:37 PM, Chet Ramey wrote:
> On 1/11/16 3:29 PM, Martin Kealey wrote:
>> Surely a pipe counts as an explicit redirection?
>
> It doesn't.

I should be clear that this is according to what the standard says. The
existing code attempts to inhibit this implicit redirection in the
presence of an input pipe.

[Martin Kealey]

Surely a pipe counts as an explicit redirection?

And even if it didn't, the inconsistency between the first and subsequent asynchronous elements is confusing.

On 12 Jan 2016 02:41, "Chet Ramey" <chet....@case.edu> wrote:

On 1/10/16 12:54 PM, Piotr Grzybowski wrote:
> hey,
>
>  I am quite sure it happens here (devel branch, at
> 6f82653c5ef09aeeeba4376a1c65ce86c3605c00):
>
> execute_cmd.c +5115:
>    if ((cmdflags & CMD_STDIN_REDIR) &&
>   pipe_in == NO_PIPE &&
>   (stdin_redirects (redirects) == 0))
>      async_redirect_stdin ();
>
>  but after reading the comments some 3.5k lines earlier I am not sure
> how to fix it; of course commenting out the if in
> execute_cmd.c:5112-5115 solves it, but who can tell what it breaks?

Posix requires this behavior:

"The standard input for an asynchronous list, before any explicit
redirections are performed, shall be considered to be assigned to a file
that has the same properties as /dev/null."

[Chet Ramey]

On 1/10/16 2:23 AM, Martin D Kealey wrote:

> Bash Version: 4.3
> Patch Level: 11
> Release Status: release
>
> Description:
> The first backgrounded element within a pipeline component has its
> stdin redirected to /dev/null; this does not apply to the second and
> subsequent background elements.
>
> On a Linux system this can be observed with:
> echo x | ( ls -ld /proc/self/fd/0 &
> ls -ld /proc/self/fd/0 &
> wait )
>
> On other systems it is apparent simply with:
> echo x | ( cat & wait )

Thanks for the report. I fixed this family of problems and the fix will
be in the next bash release, as well as the next devel snapshot.

Chet

[Martin Kealey]

On Mon, 11 Jan 2016, Chet Ramey wrote:
> "The standard input for an asynchronous list, before any explicit
> redirections are performed, shall be considered to be assigned to a file
> that has the same properties as /dev/null."

I thought the (only) point of that to avoid a background job reading the tty
in an environment that lacked job control.

The standard is actually a bit vague; under "Compound Commands" it mentions
that "Each redirection will apply to all the commands within the compound
command that do not explicitly override that redirection" but it fails to
mention that this applies to pipes as well; so I submit that it is a
reasonable interpretation that a pipe on an outer block counts as an
(explicit) redirection for the purpose of reading the "Asynchronous Lists"
sub section.

What do other shells do? Ksh? Dash? Zsh?

-Martin

[Andreas Schwab]

Martin Kealey <mar...@kurahaupo.gen.nz> writes:

> What do other shells do? Ksh? Dash? Zsh?

$ ksh -c 'printf "foo1\nfoo2\n" | { (read x; echo 1: $x) & (read x; echo 2: $x) & }'
1: foo1
2: foo2
$ dash -c 'printf "foo1\nfoo2\n" | { (read x; echo 1: $x) & (read x; echo 2: $x) & }'
1:
2:
$ zsh -c 'printf "foo1\nfoo2\n" | { (read x; echo 1: $x) & (read x; echo 2: $x) & }'
1: foo1
2: foo2
$ bash -c 'printf "foo1\nfoo2\n" | { (read x; echo 1: $x) & (read x; echo 2: $x) & }'
1:
2: foo1

[Andreas Schwab]

Martin Kealey <mar...@kurahaupo.gen.nz> writes:

> The standard is actually a bit vague; under "Compound Commands" it mentions
> that "Each redirection will apply to all the commands within the compound
> command that do not explicitly override that redirection" but it fails to
> mention that this applies to pipes as well; so I submit that it is a
> reasonable interpretation that a pipe on an outer block counts as an
> (explicit) redirection for the purpose of reading the "Asynchronous Lists"
> sub section.

I don't think so.
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07
doesn't mention pipelines.

[Martin Kealey]

On 13 Jan 2016 8:14 p.m., "Andreas Schwab" <sch...@suse.de> wrote:

> I don't think so.
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07
> doesn't mention pipelines.

That was rather my point. Given that they aren't mentioned, one has to take them as read, otherwise even ( ( foo 2>&1 ) | bar ) won't work as advertised.

[Andreas Schwab]

Martin Kealey <pg...@google.com> writes:

> On 13 Jan 2016 8:14 p.m., "Andreas Schwab" <sch...@suse.de> wrote:
>
>> I don't think so.
>>
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07
>> doesn't mention pipelines.
>
> That was rather my point. Given that they aren't mentioned, one has to take
> them as read,

What does "take them as read" mean?

[Martin Kealey]

On Wed, 13 Jan 2016, Andreas Schwab wrote:
> What does "take them as read" mean?

Taking something as read means that the author thought we would understand
it without them actually writing it down.

Or more prosaically, it means to pretend that they're written there, even
though they're not, because otherwise what's actually written is nonsense.


In this case, they mention that redirections applied to a compound statement
take effect on all its parts, but they do NOT mention that a pipe connecting
a compound statement to some other statement takes effect on its parts.

But we have to assume it does, otherwise all manner of stuff is broken, even
though they haven't written it.

-Martin

[Chet Ramey]

On 1/12/16 7:20 PM, Martin Kealey wrote:
>
>
> On Mon, 11 Jan 2016, Chet Ramey wrote:
>> "The standard input for an asynchronous list, before any explicit
>> redirections are performed, shall be considered to be assigned to a file
>> that has the same properties as /dev/null."
>
> I thought the (only) point of that to avoid a background job reading the tty
> in an environment that lacked job control.

That's the effect now, yes. The original Bourne shell, since it didn't
have job control, made no distinction, and the standard doesn't either.

> The standard is actually a bit vague; under "Compound Commands" it mentions
> that "Each redirection will apply to all the commands within the compound
> command that do not explicitly override that redirection" but it fails to
> mention that this applies to pipes as well; so I submit that it is a
> reasonable interpretation that a pipe on an outer block counts as an
> (explicit) redirection for the purpose of reading the "Asynchronous Lists"
> sub section.

Agreed, and, as I said, the existing code attempts to do that. The problem
is propagating the input pipe status into commands executed as part of a
compound command like a () subshell.

[Chet Ramey]

On 1/13/16 4:08 AM, Andreas Schwab wrote:
> Martin Kealey <mar...@kurahaupo.gen.nz> writes:
>
>> What do other shells do? Ksh? Dash? Zsh?
>
> $ ksh -c 'printf "foo1\nfoo2\n" | { (read x; echo 1: $x) & (read x; echo 2: $x) & }'
> 1: foo1
> 2: foo2

I changed bash to behave like this, which I think is the most reasonable.

[Martin Kealey]

On Wed, 13 Jan 2016, Chet Ramey wrote:
> On 1/13/16 4:08 AM, Andreas Schwab wrote:
> > Martin Kealey <mar...@kurahaupo.gen.nz> writes:
> >
> >> What do other shells do? Ksh? Dash? Zsh?
> >
> > $ ksh -c 'printf "foo1\nfoo2\n" | { (read x; echo 1: $x) & (read x; echo 2: $x) & }'
> > 1: foo1
> > 2: foo2
>
> I changed bash to behave like this, which I think is the most reasonable.

Agreed, thank you.

-Martin