Weird set -u error

[Tim Woodall]

$ cat Makefile.test
SHELL := /bin/bash -u
MAKEFLAGS += --no-builtin-rules
.SUFFIXES:

all:
echo done
$ make -f Makefile.test
echo done
done
$ make -f Makefile.test | cat
echo done
/etc/bash.bashrc: line 7: PS1: unbound variable
done
$ cat <(make -f Makefile.test)
echo done
done
$


Why do I get that unbound variable error? (I know how to fix it, I just
don't understand why it only happens when I pipe the output)

And is this a bug? In make?

The only way I've managed to reproduce this other than as above is by
explictly sourcing /etc/bash.bashrc where there are a few places where
set -u can report unbound variables.

Tim

[Thomas Schmitt]

Hi,

> $ cat Makefile.test
> SHELL := /bin/bash -u

> [...]

> /etc/bash.bashrc: line 7: PS1: unbound variable

> [...]

> Why do I get that unbound variable error?

According to a comment in /etc/bash.bashrc it tests for emptiness of PS1
in order to detect non-interactive shell sessions:

# If not running interactively, don't do anything
[ -z "$PS1" ] && return

See
https://sources.debian.org/src/bash/5.2~rc2-2/debian/etc.bash.bashrc/#L6


It should rather test for the variable being set, rather than being empty.

Jumping ahead of more skilled experts i found in the web
https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash
which brings me to a proposal for /etc/bash.bashrc

[ -z "${PS1:+x}" ] && return

Testing it:

$ bash -u

reports various "unbound variable".
Now test whether a non-empty PS1 is properly recognized

$ [ -z "${PS1:+x}" ] && echo "empty"
$

Now with undefined PS1:

$ unset PS1

makes the prompt invisible, but

[ -z "${PS1:+x}" ] && echo "empty"
empty

and

set PS1=
[ -z "${PS1:+x}" ] && echo "empty"
empty

Back to a visible prompt:

PS1='$ '
$ [ -z "${PS1:+x}" ] && echo "empty"
$


Have a nice day :)

Thomas

[to...@tuxteam.de]

On Fri, Aug 26, 2022 at 10:01:06AM +0200, Thomas Schmitt wrote:
> Hi,
>
> > $ cat Makefile.test
> > SHELL := /bin/bash -u
> > [...]
> > /etc/bash.bashrc: line 7: PS1: unbound variable
> > [...]
> > Why do I get that unbound variable error?
>
> According to a comment in /etc/bash.bashrc it tests for emptiness of PS1
> in order to detect non-interactive shell sessions:
>
> # If not running interactively, don't do anything
> [ -z "$PS1" ] && return
>
> See
> https://sources.debian.org/src/bash/5.2~rc2-2/debian/etc.bash.bashrc/#L6
>
>
> It should rather test for the variable being set, rather than being empty.
>
> Jumping ahead of more skilled experts i found in the web
> https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash
> which brings me to a proposal for /etc/bash.bashrc
>
> [ -z "${PS1:+x}" ] && return

[...]

Ah, great minds think alike :)

Why {PS1:+x} rather than {PS1:-}?

(Not a rethorical question: I think I can learn more shell programming
from you than the other way around)

Cheers
--
t

[to...@tuxteam.de]

On Fri, Aug 26, 2022 at 08:17:10AM +0100, Tim Woodall wrote:
> $ cat Makefile.test
> SHELL := /bin/bash -u
> MAKEFLAGS += --no-builtin-rules
> .SUFFIXES:
>
> all:
> echo done
> $ make -f Makefile.test
> echo done
> done
> $ make -f Makefile.test | cat
> echo done
> /etc/bash.bashrc: line 7: PS1: unbound variable

^^^^^^^^^^^

[...]

> Why do I get that unbound variable error? (I know how to fix it, I just
> don't understand why it only happens when I pipe the output)

If I were you, I'd go looking at your /etc/bashrc and see what's around
line 7. Mine says (line numbers added by me):

6 # If not running interactively, don't do anything
7 [ -z "$PS1" ] && return

...and indeed, bash -u complains there, because it tries to expand an
(unset) PS1 before checking it for emptyness.

> And is this a bug? In make?

Arguably it's a bug(let) in /etc/bash.bashrc. This would work better:

[ -z "${PS1:-}" ] && return

But I'll defer to more shell-savvy people for a proper fix.

> The only way I've managed to reproduce this other than as above is by
> explictly sourcing /etc/bash.bashrc where there are a few places where
> set -u can report unbound variables.

I think this /is/ what you are seeing.

Cheers
--
t

[Tim Woodall]

On Fri, 26 Aug 2022, to...@tuxteam.de wrote:

> On Fri, Aug 26, 2022 at 08:17:10AM +0100, Tim Woodall wrote:
>> $ cat Makefile.test
>> SHELL := /bin/bash -u
>> MAKEFLAGS += --no-builtin-rules
>> .SUFFIXES:
>>
>> all:
>> echo done
>> $ make -f Makefile.test
>> echo done
>> done
>> $ make -f Makefile.test | cat
>> echo done
>> /etc/bash.bashrc: line 7: PS1: unbound variable
> ^^^^^^^^^^^
>
> [...]
>
>> Why do I get that unbound variable error? (I know how to fix it, I just
>> don't understand why it only happens when I pipe the output)
>
> If I were you, I'd go looking at your /etc/bashrc and see what's around
> line 7. Mine says (line numbers added by me):

Yes, I know how to fix it. That is straight forward. SUDO_USER and
SUDO_PS1 have similar issues, debian_chroot works in the presence of set
-u

But I don't understand why /etc/bash.bashrc is being sourced by make
when piping make through cat. Make is doing something like:

source /etc/bash.bashrc; bash -c 'echo done' - but only when the output
is to a pipe.

bash -uc 'echo done'
doesn't have the problem whether or not it is outputting to a pipe.

Tim

[to...@tuxteam.de]

On Fri, Aug 26, 2022 at 09:19:03AM +0100, Tim Woodall wrote:

[...]

> Yes, I know how to fix it. That is straight forward. SUDO_USER and
> SUDO_PS1 have similar issues, debian_chroot works in the presence of set
> -u

I see.

> But I don't understand why /etc/bash.bashrc is being sourced by make
> when piping make through cat. Make is doing something like:

The shell must be thinking that it is interactive. I'm a bit pressed at
the moment, so I'll have to leave the "why" as an exercise... you know
:)

Cheers
--
t

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> Why {PS1:+x} rather than {PS1:-}?

It seemed to be the nearest suitable variation which was similar to the
proposal
[ ${var+x} ]
in
https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash
where one can see a nice table of ${VARIABLE...} tricks:
parameter expansion VARIABLE set VARIABLE empty VARIABLE unset
...
>From the hint that it is defined by POSIX i got to the table at
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
under "2.6.2 Parameter Expansion".


As stated at other occasions, my shell mindset is still from the late 1980s
when the choice was Bourne shell or C shell. When newer skills are needed,
i search the web or look into my own existing scripts.

[to...@tuxteam.de]

On Fri, Aug 26, 2022 at 11:04:25AM +0200, Thomas Schmitt wrote:
> Hi,
>
> to...@tuxteam.de wrote:
> > Why {PS1:+x} rather than {PS1:-}?
>
> It seemed to be the nearest suitable variation which was similar to the
> proposal
> [ ${var+x} ]

[...]

Thanks :)

Cheers
--
t

[David]

On Fri, 26 Aug 2022 at 18:02, <to...@tuxteam.de> wrote:
> On Fri, Aug 26, 2022 at 08:17:10AM +0100, Tim Woodall wrote:

> > $ cat Makefile.test
> > SHELL := /bin/bash -u
> > MAKEFLAGS += --no-builtin-rules
> > .SUFFIXES:
> >
> > all:
> > echo done
> > $ make -f Makefile.test
> > echo done
> > done
> > $ make -f Makefile.test | cat
> > echo done
> > /etc/bash.bashrc: line 7: PS1: unbound variable

> ^^^^^^^^^^^

> > Why do I get that unbound variable error? (I know how to fix it, I just
> > don't understand why it only happens when I pipe the output)

Hi, I'm no expert, but I would explain this as follows.

> > $ make -f Makefile.test

In the above command, make runs an interactive shell, because its standard
output is connected to a terminal.

> > $ make -f Makefile.test | cat

Whereas in the above command, make runs a non-interactive shell, because
it is in a pipeline so its standard output is not connected to a terminal,
but rather to the standard input of the 'cat' command.

PS1 is unbound in the second case because when the shell starts in
non-interactive mode, it does not set PS1.

The behaviour of interactive shells is explained in the three pages
linked here:
https://www.gnu.org/software/bash/manual/html_node/Interactive-Shells.html

You can find expert responses on this topic by asking the mailing list:
https://lists.gnu.org/mailman/listinfo/help-bash

> [ -z "${PS1:-}" ] && return
>
> But I'll defer to more shell-savvy people for a proper fix.

Regarding the side discussion of how to test whether a bash shell is
interactive, both bash and dash shells provide the 'i' flag inside the
special variable $- (hyphen) for precisely this purpose.

So I would be inclinded to use that, rather than the old way of examining
$PS1. Unlikely, but possible, that some bizarre user has unset their $PS1
variable.

Some bash project documentation on this topic is here:
https://www.gnu.org/software/bash/manual/html_node/Is-this-Shell-Interactive_003f.html

[Tim Woodall]

Thanks! I'd been trying -l but

$ bash -uic 'echo done'
bash: SUDO_USER: unbound variable
bash: color_prompt: unbound variable
done
$

Interesting that make is creating an interactive shell when its output
is redirected!

Tim.

[Greg Wooledge]

On Fri, Aug 26, 2022 at 08:17:10AM +0100, Tim Woodall wrote:
> $ cat Makefile.test
> SHELL := /bin/bash -u

WHY?!

> Why do I get that unbound variable error?

Because you turned on set -u in a place where the code had not been
adjusted to account for set -u's foibles.

set -u breaks scripts. It's not as bad as set -e (nothing is that bad),
but it's still a dramatic change to the shell's behavior, and some
perfectly legitimate code *will* break.

On Fri, Aug 26, 2022 at 10:01:06AM +0200, Thomas Schmitt wrote:
> According to a comment in /etc/bash.bashrc it tests for emptiness of PS1
> in order to detect non-interactive shell sessions:
>

> # If not running interactively, don't do anything

> [ -z "$PS1" ] && return

>
> It should rather test for the variable being set, rather than being empty.

If you're testing whether a shell is interactive, the *proper* way is
to check for the presence of "i" in the "$-" special parameter:

case $- in *i*) :;; *) return;; esac

Or, bash only:

[[ $- = *i* ]] || return

Testing whether PS1 is set is a *hack* and it will fail (giving false
positives) if the user has exported the PS1 variable. Which some people
do.

> $ [ -z "${PS1:+x}" ] && echo "empty"

The ":" gives a positive result if the variable is defined but empty,
which may not be what you want. Dunno, hard to guess why anyone would
want an empty PS1 variable, but I suppose it's possible. People do weird
stuff all the time.

On Fri, Aug 26, 2022 at 11:04:25AM +0200, Thomas Schmitt wrote:

> to...@tuxteam.de wrote:
> > Why {PS1:+x} rather than {PS1:-}?
>
> It seemed to be the nearest suitable variation which was similar to the
> proposal
> [ ${var+x} ]

You *should* still use quotes here. Yeah, you might be thinking, the
only possible expansions are the empty string or "x", and it handles
both of those correctly, but it's still a matter of principle and good
habits. By omitting the quotes, you're forcing the shell to perform
word splitting and globbing steps on the result, which actually makes
it run more slowly, even when those steps don't change the result.

[Greg Wooledge]

On Fri, Aug 26, 2022 at 11:14:37AM +0100, Tim Woodall wrote:
> $ bash -uic 'echo done'
> bash: SUDO_USER: unbound variable
> bash: color_prompt: unbound variable
> done
> $

Cute. Mine's even weirder:

unicorn:~$ bash -uic 'echo done'
bash: SUDO_USER: unbound variable
bash: no: unbound variable
done

It turns out, this is due to the following line in my .bashrc:

MAILCHECK=no

I can't even remember putting that in there. Here's what the man page
says:

MAILCHECK
Specifies how often (in seconds) bash checks for mail. The de‐
fault is 60 seconds. When it is time to check for mail, the
shell does so before displaying the primary prompt. If this
variable is unset, or set to a value that is not a number
greater than or equal to zero, the shell disables mail checking.

Apparently "not a number greater than or equal to zero" must not mean
what I thought it meant, at the time I wrote that line.

At some point, bash must have given MAILCHECK the "integer attribute",
which means that MAILCHECK=no is attempting assignment to an *arithmetic*
variable, which means it performs a recursive expansion on the word "no"
in an arithmetic context -- which makes it treat "no" as a shell variable
to be expanded. Hence the complaint from -u.

Without -u, and without a "no" variable, the effect is basically
MAILCHECK=0 which is clearly not what I wanted.

Fortunately, it's entirely moot, because in context, I have this:

MAILCHECK=no
unset MAILCHECK # Takes too damned long.

I guess I should just comment out the first line, then. There's no point
keeping it there.

So, today I guess I learned that MAILCHECK gained the integer attribute
at some point. I wonder when... well, let's find out.

...
unicorn:~$ bash-3.1 -uic :
bash-3.1: no: unbound variable
unicorn:~$ bash-3.0 -uic :
unicorn:~$

Looks like it happened in bash 3.1.

*Now* I'll comment out that line in my .bashrc.

[to...@tuxteam.de]

On Fri, Aug 26, 2022 at 07:24:44AM -0400, Greg Wooledge wrote:
> On Fri, Aug 26, 2022 at 08:17:10AM +0100, Tim Woodall wrote:
> > $ cat Makefile.test
> > SHELL := /bin/bash -u
>
> WHY?!
>
> > Why do I get that unbound variable error?
>
> Because you turned on set -u in a place where the code had not been
> adjusted to account for set -u's foibles.
>
> set -u breaks scripts. It's not as bad as set -e (nothing is that bad),
> but it's still a dramatic change to the shell's behavior, and some
> perfectly legitimate code *will* break.

Yeah, but the interesting question still remaining is why a
call to a shell from a makefile is interactive in the first
place (otherwise it wouldn't be sourcing /etc/bash.bashrc,
which is what's triggering the error).

Perhaps an academic exercise, I don't know.

[...]

> > [ -z "$PS1" ] && return
> >
> > It should rather test for the variable being set, rather than being empty.
>
> If you're testing whether a shell is interactive, the *proper* way is
> to check for the presence of "i" in the "$-" special parameter:
>
> case $- in *i*) :;; *) return;; esac
>
> Or, bash only:
>
> [[ $- = *i* ]] || return

Note that this "you" doesn't include the OP: the problematic test is
in /etc/bash.bashrc, which is Debian-provided code.

> Testing whether PS1 is set is a *hack* and it will fail (giving false
> positives) if the user has exported the PS1 variable. Which some people
> do.
>
> > $ [ -z "${PS1:+x}" ] && echo "empty"
>
> The ":" gives a positive result if the variable is defined but empty,
> which may not be what you want. Dunno, hard to guess why anyone would
> want an empty PS1 variable, but I suppose it's possible. People do weird
> stuff all the time.

---

> On Fri, Aug 26, 2022 at 11:04:25AM +0200, Thomas Schmitt wrote:
> > to...@tuxteam.de wrote:
> > > Why {PS1:+x} rather than {PS1:-}?
> >
> > It seemed to be the nearest suitable variation which was similar to the
> > proposal
> > [ ${var+x} ]
>
> You *should* still use quotes here. Yeah, you might be thinking, the
> only possible expansions are the empty string or "x", and it handles
> both of those correctly, but it's still a matter of principle and good
> habits. By omitting the quotes, you're forcing the shell to perform
> word splitting and globbing steps on the result, which actually makes
> it run more slowly, even when those steps don't change the result.

To be fair, Thomas's recommendation, as well as mine both had the
quotes. Thomas was just quoting (heh) where he got his version from.

But all in all we're still "fixing" a package-provided script. Should
we file a bug against it? WDYT?

And then, there's that other riddle: why is a shell invoked from a
Makefile sourcing /etc/bash.bashrc in the first place?

Cheers
--
t

[David]

On Fri, 26 Aug 2022 at 22:14, <to...@tuxteam.de> wrote:

> And then, there's that other riddle: why is a shell invoked from a
> Makefile sourcing /etc/bash.bashrc in the first place?

Hi, my understanding is that the rule commands in the Makefile are run by
a shell, one for each line. The output stream of those commands is usually
stdout (unless redirected), to allow the user to see the output. The shell
detects that the command output is not redirected away from stdout, and so
it starts itself in interactive mode.

'man bash' says:

When an interactive shell that is not a login shell is started, bash
reads and executes commands from /etc/bash.bashrc and ~/.bashrc, if
these files exist. This may be inhibited by using the --norc option.

[David]

On Fri, 26 Aug 2022 at 22:14, <to...@tuxteam.de> wrote:
> On Fri, Aug 26, 2022 at 07:24:44AM -0400, Greg Wooledge wrote:
> > On Fri, Aug 26, 2022 at 08:17:10AM +0100, Tim Woodall wrote:

[...]

> > On Fri, Aug 26, 2022 at 11:04:25AM +0200, Thomas Schmitt wrote:
> > > to...@tuxteam.de wrote:

> But all in all we're still "fixing" a package-provided script. Should
> we file a bug against it? WDYT?

In my opinion, the only "bug" is the user putting -u in the SHELL
definition contained in the Makefile. I haven't yet seen them
answer Greg's enquiry=WHY?!

[Tim Woodall]

But that's backwards, make is starting an interactive shell when its
output is redirected. When I don't redirect make's output it works.


>

[Tim Woodall]

On Fri, 26 Aug 2022, David wrote:

I disagree with Greg.

The real Makefile has -euo pipefail - and works perfectly, even from
cron. I tend to use this in all scripts. -u in particular is useful for
picking up typos and -e makes it fail fast and early.

It's only when I attempted to redirect output that I spotted this.

The failure doesn't even break anything, it just creates noise on
stderr.

I might log a bug to fix /etc/bash.bashrc but I'm suspecting there's a
bug in make - and changing .bashrc will just hide the symptoms.

Tim.

[Greg Wooledge]

On Fri, Aug 26, 2022 at 02:40:55PM +0100, Tim Woodall wrote:
> But that's backwards, make is starting an interactive shell when its
> output is redirected. When I don't redirect make's output it works.

I thought you were piping in the original message. There is a difference
between a pipe, and a redirection.

You might want to test both, and see exactly which one(s) trigger this
behavior. It's still not clear to me whether it's make(1) passing the
-i option for some reason, or bash incorrectly deciding that it should
read these startup files, based on whatever criteria it uses to sense
whether it's being run as an interactive shell.

There are also various hacks that are compiled into Debian's version
of bash, for example the one that reads the interactive startup files
if bash believes it's the child of an sshd daemon. This is not the
default behavior of upstream bash, but most Linux distributions enable
this behavior because they believe, rightly or wrongly, that their users
bases expect it.

At this moment, I'm kind of leaning toward one of those hacks being
triggered by your specific combination of factors.

[David]

As already stated, my guess is that while you might be redirecting the
make command line, there are no redirection operations in the commands
in the Makefile rules, so the the shells started by make are interactive.

Anyway, as I already suggested, if you ask your original question to
a mailing list of either the 'make' project [1] or the 'bash' project [2], then
the developers of these projects can give you a more authoritative answer
than the speculations of random Debian users such as myself.

[1] https://savannah.gnu.org/mail/?group=make
[2] https://savannah.gnu.org/mail/?group=bash

[Greg Wooledge]

On Fri, Aug 26, 2022 at 02:51:47PM +0100, Tim Woodall wrote:
> The real Makefile has -euo pipefail

I have stopped reading and caring now. Good luck.

[to...@tuxteam.de]

On Fri, Aug 26, 2022 at 10:57:44AM -0400, Greg Wooledge wrote:

[...]

> There are also various hacks that are compiled into Debian's version

> of bash [...]

> At this moment, I'm kind of leaning toward one of those hacks being
> triggered by your specific combination of factors.

That's the most convincing conjecture so far, I'd say :)

Cheers
--
t

[Tim Woodall]

It's bash:

$ bash -uc :
$ ( bash -uc : )
'PS1='
/etc/bash.bashrc: line 8: PS1: unbound variable
$ ( bash -uc : ; : )
$

I modified /etc/bash.bashrc to prove it was only being sourced in the
second case above (it prints the value of PS1)

And now I can reproduce without -u
$ bash -c :
$ ( bash -c : )
'PS1='
$ ( bash -c : ; : )
$

[Greg Wooledge]

On Fri, Aug 26, 2022 at 11:54:06PM +0100, Tim Woodall wrote:
> I modified /etc/bash.bashrc to prove it was only being sourced in the
> second case above (it prints the value of PS1)
>
> And now I can reproduce without -u
> $ bash -c :
> $ ( bash -c : )
> 'PS1='
> $ ( bash -c : ; : )
> $

I can't reproduce this on Debian 11, with bash 5.1-2+deb11u1.
What version of bash is this?

[Greg Wooledge]

On Sat, Aug 27, 2022 at 10:21:08AM +1000, David wrote:
> Just sharing something that I tried, to shed some more light
> on this, in case that's useful ...
>
> $ foo() { case "$-" in *i* ) i=interactive ;; * ) i=not-interactive ;;
> esac ; echo $BASHPID $i ; }
> $ declare -fx foo
> $ foo
> 2415 interactive
> $ foo ; ( foo ) ; foo
> 2415 interactive
> 2512 interactive
> 2415 interactive
> $ bash -c foo
> 2515 not-interactive
> $ foo ; ( bash -c foo ) ; foo
> 2415 interactive
> 2516 not-interactive
> 2415 interactive
> $ foo ; ( foo ; bash -c foo ; foo ) ; foo
> 2415 interactive
> 2517 interactive
> 2518 not-interactive
> 2517 interactive
> 2415 interactive

I get these results as well, with Debian 11's packaged bash.

Has anyone managed to reproduce the OP's results, either getting
"interactive" from a bash -c call, or seeing *any* evidence that
/etc/bash.bashrc or ~/.bashrc is sourced from a bash -c call?

[David]

Just sharing something that I tried, to shed some more light
on this, in case that's useful ...

$ foo() { case "$-" in *i* ) i=interactive ;; * ) i=not-interactive ;;
esac ; echo $BASHPID $i ; }
$ declare -fx foo
$ foo
2415 interactive
$ foo ; ( foo ) ; foo
2415 interactive
2512 interactive
2415 interactive
$ bash -c foo
2515 not-interactive
$ foo ; ( bash -c foo ) ; foo
2415 interactive
2516 not-interactive
2415 interactive
$ foo ; ( foo ; bash -c foo ; foo ) ; foo
2415 interactive
2517 interactive
2518 not-interactive
2517 interactive
2415 interactive

$

[David]

Yeah, sorry, I forgot to include, for the record:

$ echo $BASH_VERSION
5.1.4(1)-release
$ apt list --installed bash
Listing... Done
bash/stable,now 5.1-2+deb11u1 amd64 [installed]

[David]

On Sat, 27 Aug 2022 at 10:27, Greg Wooledge <gr...@wooledge.org> wrote:

> Has anyone managed to reproduce the OP's results, either getting
> "interactive" from a bash -c call, or seeing *any* evidence that
> /etc/bash.bashrc or ~/.bashrc is sourced from a bash -c call?

On Debian 11, when I create a test user, login on a console as that
user, and duplicate the recipe provided in the original message[1],
the reported problem does NOT occur:

t220827a@kablamm:~$ cat Makefile.test
SHELL := /bin/bash -u

MAKEFLAGS += --no-builtin-rules
.SUFFIXES:

all:
echo done

t220827a@kablamm:~$ make -f Makefile.test
echo done
done
t220827a@kablamm:~$ make -f Makefile.test | cat
echo done
done
t220827a@kablamm:~$ cat <(make -f Makefile.test)
echo done
done
t220827a@kablamm:~$

[1] https://lists.debian.org/debian-user/2022/08/msg00578.html

[gene heskett]

> .
Why the miss-match, I get the same results as this myself.

Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/>

[to...@tuxteam.de]

Same results here...

> Yeah, sorry, I forgot to include, for the record:
>
> $ echo $BASH_VERSION
> 5.1.4(1)-release
> $ apt list --installed bash
> Listing... Done
> bash/stable,now 5.1-2+deb11u1 amd64 [installed]

... and same bas version. Hmmm.

Cheers
--
t

[to...@tuxteam.de]

On Sat, Aug 27, 2022 at 11:22:09AM +1000, David wrote:
> On Sat, 27 Aug 2022 at 10:27, Greg Wooledge <gr...@wooledge.org> wrote:
>
> > Has anyone managed to reproduce the OP's results, either getting
> > "interactive" from a bash -c call, or seeing *any* evidence that
> > /etc/bash.bashrc or ~/.bashrc is sourced from a bash -c call?
>
> On Debian 11, when I create a test user, login on a console as that
> user, and duplicate the recipe provided in the original message[1],
> the reported problem does NOT occur:

Aha...

so main suspects are now ~/.bash_profile ~/.inputrc or some exported
environment lying around...

Cheers
--
t

[David Wright]

On Sat 27 Aug 2022 at 00:23:10 (-0400), gene heskett wrote:
> On 8/26/22 20:35, David wrote:
> > On Sat, 27 Aug 2022 at 10:27, Greg Wooledge <gr...@wooledge.org> wrote:
> > > I get these results as well, with Debian 11's packaged bash.
> > Yeah, sorry, I forgot to include, for the record:
> >
> > $ echo $BASH_VERSION
> > 5.1.4(1)-release
> > $ apt list --installed bash
> > Listing... Done
> > bash/stable,now 5.1-2+deb11u1 amd64 [installed]
> >
> > .
> Why the miss-match, I get the same results as this myself.

bash_5.1-2+deb11u1_amd64.deb

bash is package name

5.1 is upstream version

2+deb11u1 is /Debian/ version

amd64 is architecture.

Cheers,
David.

[David]

On Sat, 27 Aug 2022 at 08:54, Tim Woodall <debia...@woodall.me.uk> wrote:
> On Fri, 26 Aug 2022, to...@tuxteam.de wrote:
> > On Fri, Aug 26, 2022 at 10:57:44AM -0400, Greg Wooledge wrote:

> >> There are also various hacks that are compiled into Debian's version
> >> of bash [...]

> >> At this moment, I'm kind of leaning toward one of those hacks being
> >> triggered by your specific combination of factors.

> > That's the most convincing conjecture so far, I'd say :)

> It's bash:

> $ bash -uc :
> $ ( bash -uc : )
> 'PS1='
> /etc/bash.bashrc: line 8: PS1: unbound variable
> $ ( bash -uc : ; : )

I am still thinking about this weirdness, as a learning exercise :)

Some observations and conjecture:

1. No-one except Tim is yet able to reproduce the above. Tim has not given
us enough information to do so.

2. We dont know what shell and version is initially running *before* any of
the above occurs. That information is essential. It could be 'bash' or
'dash' or anything else with different behaviour we have not yet
considered.

3. It looks like the initial shell is being used interactively, and has
PS1='$ ' which suggests that is not 'bash', possibly 'dash'.

4. The ':' command is a shell builtin, not an external command. That could
be a relevant fact in the above example.

5. My 'foo' examples in a previous message demonstrate that bash -c ... is
always non-interactive.

6. 'man bash' tells us that bash when started interactively includes the
steps of: (a) set PS1 (b) source /etc/bash.bashrc

7. But the Makefile example originally given by Tim requires that (b)
occurs as if (a) never happened.

8. Similarly the above error message requires that (b) occurs as if (a)
never happened.

9. How is this error possible then, if an interactive shell does both (a)
and (b) when it starts? What could possibly be unsetting PS1 before
step (b) occurs. Something that could occur between (a) and (b) could
be any commands inside a file named in either an ENV or BASH_ENV
environment variable. We need to know if either of these exist.

10. On the other hand, a non-interactive shell could do (b) without (a) if
BASH_ENV=/etc/bash.bashrc was set in the environment that it inherits.

11. Regarding this line from above:

$ ( bash -uc : )

I could attempt to explain what happens as: the initial shell forks
a subshell clone of itself because of the parentheses. That subshell
clone is interactve. It somehow has PS1 unset, and stupidly sources
/etc/bash.bashrc which gives the error. The subshell then runs
'bash -uc :' as a non-interactive shell.

That might make sense, except for the fact that when I run
that example here, I do not see any errors!

Of course I don't want to see errors, but the lack of them indicates
that my shell and everyone elses shell don't work in the way that my
explanation says they do.

Perhaps, whatever strange shell Tim has does source /etc/bash.bashrc
when it forks a subshell environment, but my shell does not. I don't
see a good reason why any subshell would source that file again,
because its environment is supposed to be a clone of its parent.

12. And regarding this line from above:

$ ( bash -uc : ; : )

Maybe Tim's strange shell does not error on this line, because
the command is not a simple builtin. I have no clue.

Finally, to take this any further, I think we need to know:

What the initial shell is.
The output of 'env' in the initial shell.
(particularly any ENV or BASH_ENV values)
The output of 'cat ~/.bashrc'

[Tim Woodall]

This just gets weirder and weirder.

It looks like it's related to logging in with ssh:

I am running a modified ssh. It's been patched to allow cipher None and
I've backported from sid

I had to jump through some hoops to login at the console but now:

aptmirror17 login: apt-mirror
Password:
Linux aptmirror17.home.woodall.me.uk 5.10.0-16-amd64 #1 SMP Debian
5.10.127-2 (2022-07-23) x86_64

The programs included with the Debian GNU/Linux system are free
software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Aug 27 06:22:26 UTC 2022 on hvc0
sourcing /etc/bash.bashrc
apt-mirror@aptmirror17:~$ ( bash -cu : )
apt-mirror@aptmirror17:~$


apt-mirror@aptmirror17:~$ ssh aptmirror17
Enter passphrase for key '/var/spool/apt-mirror/.ssh/id_rsa':
Linux aptmirror17.home.woodall.me.uk 5.10.0-16-amd64 #1 SMP Debian
5.10.127-2 (2022-07-23) x86_64

The programs included with the Debian GNU/Linux system are free
software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Aug 27 06:23:12 2022
apt-mirror@aptmirror17:~$ ( bash -cu : )

/etc/bash.bashrc: line 7: PS1: unbound variable

apt-mirror@aptmirror17:/mnt/mirror/local-debs$ ssh -V
OpenSSH_9.0p1 Debian-1+~tjw11r1, OpenSSL 1.1.1n 15 Mar 2022



Downgrading ssh back to the version from bullseye shows it's not my
local changes:

apt-mirror@aptmirror17:~$ ssh aptmirror17
Enter passphrase for key '/var/spool/apt-mirror/.ssh/id_rsa':
Linux aptmirror17.home.woodall.me.uk 5.10.0-16-amd64 #1 SMP Debian
5.10.127-2 (2022-07-23) x86_64

The programs included with the Debian GNU/Linux system are free
software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Aug 27 06:36:10 2022 from
2001:8b0:bfcd:100:216:3eff:fee0:7107
apt-mirror@aptmirror17:~$ ( bash -cu : )

/etc/bash.bashrc: line 7: PS1: unbound variable

apt-mirror@aptmirror17:~$

apt-mirror@aptmirror17:~$ ssh -V
OpenSSH_8.4p1 Debian-5+deb11u1, OpenSSL 1.1.1n 15 Mar 2022
apt-mirror@aptmirror17:~$

[to...@tuxteam.de]

On Sat, Aug 27, 2022 at 07:49:17AM +0100, Tim Woodall wrote:
> On Sat, 27 Aug 2022, to...@tuxteam.de wrote:
>
> > On Sat, Aug 27, 2022 at 11:22:09AM +1000, David wrote:
> > > On Sat, 27 Aug 2022 at 10:27, Greg Wooledge <gr...@wooledge.org> wrote:
> > >
> > > > Has anyone managed to reproduce the OP's results, either getting
> > > > "interactive" from a bash -c call, or seeing *any* evidence that
> > > > /etc/bash.bashrc or ~/.bashrc is sourced from a bash -c call?
> > >
> > > On Debian 11, when I create a test user, login on a console as that
> > > user, and duplicate the recipe provided in the original message[1],
> > > the reported problem does NOT occur:
> >
> > Aha...
> >
> > so main suspects are now ~/.bash_profile ~/.inputrc or some exported
> > environment lying around...
> >
> > Cheers
> >
>
> This just gets weirder and weirder.
>
> It looks like it's related to logging in with ssh:

This could be a giveaway. I tried David's test from upthread [1]
while logged in through SSH and the results are (still) the same
as his. But... my SSH is vanilla Buster, not patched, so...

[...]

> Downgrading ssh back to the version from bullseye shows it's not my
> local changes:

...but then, this makes it the more interesting :-)

Have you tried David's test?

Cheers

[1] Message-ID: <CAMPXz=pncfSExnKe7R7UaWVWK+0_...@mail.gmail.com>
--
t

[David]

On Sat, 27 Aug 2022 at 16:49, Tim Woodall <debia...@woodall.me.uk> wrote:

> This just gets weirder and weirder.
>
> It looks like it's related to logging in with ssh:

[..]

> apt-mirror@aptmirror17:~$ ( bash -cu : )
> apt-mirror@aptmirror17:~$

> apt-mirror@aptmirror17:~$ ssh aptmirror17

> Last login: Sat Aug 27 06:23:12 2022

> apt-mirror@aptmirror17:~$ ( bash -cu : )
> /etc/bash.bashrc: line 7: PS1: unbound variable

Interesting! I can reproduce this, not sure why.

I have modified my ssh environment so that it is not the default.
But I do see a similar error message, using ssh from stable Debian 11.

[david@kablamm ~]$ ( bash -cu : )
[david@kablamm ~]$ ssh kablamm
Linux kablamm 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) x86_64
Last login: Sat Aug 27 17:00:13 2022 from 10.1.1.2
[david@kablamm ~]$ ( bash -cu : )

/etc/bash.bashrc: line 7: PS1: unbound variable

[david@kablamm ~]$

So hopefully someone who understands how ssh interactive
session interacts with bash can explain this to us now :)

[Greg Wooledge]

5.1.4 is upstream's notation for bash 5.1 with the first 4 (upstream)
patches applied. The (1) means it was the first time bash had been
built in that directory.

If you work with upstream bash source, you may end up with something
like 5.1.4(2)-release meaning you compiled it once, changed something,
then compiled it again. Applying upstream patches in the same directory
might give you 5.1.5(3)-release and so on. Although typically you'll
apply more than one patch at a time, because Chet Ramey tends to release
them in groups.

[Greg Wooledge]

On Sat, Aug 27, 2022 at 05:12:57PM +1000, David wrote:
> I have modified my ssh environment so that it is not the default.
> But I do see a similar error message, using ssh from stable Debian 11.
>
> [david@kablamm ~]$ ( bash -cu : )
> [david@kablamm ~]$ ssh kablamm
> Linux kablamm 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) x86_64
> Last login: Sat Aug 27 17:00:13 2022 from 10.1.1.2
> [david@kablamm ~]$ ( bash -cu : )
> /etc/bash.bashrc: line 7: PS1: unbound variable
> [david@kablamm ~]$

Same here, with no special trickery.


unicorn:~$ (bash -cu :)
unicorn:~$ ssh localhost
greg@localhost's password:
Linux unicorn 5.10.0-17-amd64 #1 SMP Debian 5.10.136-1 (2022-08-13) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Last login: Tue Aug 16 07:35:44 2022
unicorn:~$ (bash -cu :)

/etc/bash.bashrc: line 7: PS1: unbound variable

So it's definitely coming from the Debian bash patch that triggeres the
special behavior when bash detects itself to be a child of ssh. I've
got no clue (yet!) why it's only triggered using a subshell.

I also can't explain this (still inside the ssh localhost session):


unicorn:~$ (bash -c 'declare -p PS1')
declare -- PS1="\\h:\\w\\\$ "
unicorn:~$ (bash -cu 'declare -p PS1')

/etc/bash.bashrc: line 7: PS1: unbound variable

/etc/bash.bashrc: line 1: declare: PS1: not found


Why would adding the -u option change whether PS1 gets unset? Not a clue.

Anyway, this is the segment of the upstream bash source (config-top.h)
where the compile-time ssh detection feature can be enabled:


/* Define this if you want bash to try to check whether it's being run by
sshd and source the .bashrc if so (like the rshd behavior). This checks
for the presence of SSH_CLIENT or SSH2_CLIENT in the initial environment,
which can be fooled under certain not-uncommon circumstances. */
/* #define SSH_SOURCE_BASHRC */


One of the Debian patches removes the comments around that preprocessor
variable. And here in shell.c is the function where it gets used:


static void
run_startup_files ()
{
#if defined (JOB_CONTROL)
int old_job_control;
#endif
int sourced_login, run_by_ssh;

/* get the rshd/sshd case out of the way first. */
if (interactive_shell == 0 && no_rc == 0 && login_shell == 0 &&
act_like_sh == 0 && command_execution_string)
{
#ifdef SSH_SOURCE_BASHRC
run_by_ssh = (find_variable ("SSH_CLIENT") != (SHELL_VAR *)0) ||
(find_variable ("SSH2_CLIENT") != (SHELL_VAR *)0);
#else
run_by_ssh = 0;
#endif

/* If we were run by sshd or we think we were run by rshd, execute
~/.bashrc if we are a top-level shell. */
if ((run_by_ssh || isnetconn (fileno (stdin))) && shell_level < 2)
{
#ifdef SYS_BASHRC
# if defined (__OPENNT)
maybe_execute_file (_prefixInstallPath(SYS_BASHRC, NULL, 0), 1);
# else
maybe_execute_file (SYS_BASHRC, 1);
# endif
#endif
maybe_execute_file (bashrc_file, 1);
return;
}
}
[...]


That SYS_BASHRC thing is /etc/bash.bashrc in Debian (another one of the
Debian patches turns this on -- it's not enabled by default in upstream
bash).

Now, that "shell_level < 2" check is also intriguing.

Starting from a plain old regular urxvt running bash in my local X session
which was launched by startx:


unicorn:~$ declare -p SHLVL
declare -x SHLVL="2"
unicorn:~$ ssh localhost
greg@localhost's password:
[...]
unicorn:~$ declare -p SHLVL
declare -x SHLVL="1"
unicorn:~$ bash -uc 'declare -p SHLVL'
declare -x SHLVL="2"
unicorn:~$ (bash -uc 'declare -p SHLVL')

/etc/bash.bashrc: line 7: PS1: unbound variable

declare -x SHLVL="1"


It looks like the subshell resets the shell_level, which is why the
behavior in run_startup_files() changes.

I still don't know how -u is involved (why the behavior of (bash -uc)
is different from that of (bash -c)). I'll leave that analysis to
someone who actually cares about -u.

In any event, it's the confluence of FIVE factors here that matters:

1) Using a version of bash that was compiled with SSH_SOURCE_BASHRC.
2) Being in an ssh session, so that SSH_CLIENT is set.
3) Using () to force a subshell which resets the shell_level.
4) Using -u which somehow unsets PS1.
5) Using code that checks the value of $PS1 in a startup file.

Remove any one of those factors, and the problem doesn't manifest.

[Greg Wooledge]

On Sat, Aug 27, 2022 at 09:52:11AM -0400, Greg Wooledge wrote:
> I also can't explain this (still inside the ssh localhost session):
>
>
> unicorn:~$ (bash -c 'declare -p PS1')
> declare -- PS1="\\h:\\w\\\$ "
> unicorn:~$ (bash -cu 'declare -p PS1')
> /etc/bash.bashrc: line 7: PS1: unbound variable
> /etc/bash.bashrc: line 1: declare: PS1: not found
>
>
> Why would adding the -u option change whether PS1 gets unset? Not a clue.

It turns out, this is a bit of an illusion. It's not -u causing PS1 to
be unset. bash -c unsets PS1 (because it's launching a non-interactive
shell) -- but since it's inside an ssh session and has shell_level < 2,
it *also* reads the startup files.

In the first example above, reading the startup files succeeds, because
there is no -u.

In the second example above, reading the startup files fails, because
of -u. Thus, PS1 remains unset. The part of the startup files that
would have set PS1 never gets executed.

I believe all of the behaviors have been explained now.

[David]

Thanks for sharing your explanation, that's quite an impressive
investigation. My C skills are pretty minimal so I would have no hope to to
figure any of that out. Although I can follow the syntax at the statement
level and have occasionally written or patched some useful C code myself,
I struggle to comprehend other people's real-world C at any higher level.

I think there might be one remaining aspect still mysterious, so there
might be yet another factor beyond the FIVE you identified ...

The fact that this statement, first shown by Tim, does NOT error:

$ ( bash -uc : ; : )

[david@kablamm ~]$ ssh kablamm
Linux kablamm 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) x86_64

Last login: Sat Aug 27 17:02:53 2022 from 10.1.1.2
[david@kablamm ~]$ ( bash -cu : )

/etc/bash.bashrc: line 7: PS1: unbound variable

[Greg Wooledge]

On Sun, Aug 28, 2022 at 12:46:17AM +1000, David wrote:
> I think there might be one remaining aspect still mysterious, so there
> might be yet another factor beyond the FIVE you identified ...
>
> The fact that this statement, first shown by Tim, does NOT error:
>
> $ ( bash -uc : ; : )

(inside an ssh session)

This is related to how bash optimizes subshell commands when it can.

unicorn:~$ (bash -uc :)

/etc/bash.bashrc: line 7: PS1: unbound variable

unicorn:~$ (bash -uc : ; :)
unicorn:~$

In the first one, there's just one command inside the subshell, so bash
can replace one of the fork() calls, and treat it like

(exec bash -uc :)

In the second one, there's a second command, so bash can't perform that
same optimization.

(exec bash -uc : ; :) would not work the same as (bash -uc : ; :)

This optimization affects whether the shell_level is reset:

unicorn:~$ (bash -c 'declare -p SHLVL')
declare -x SHLVL="1"
unicorn:~$ (bash -c 'declare -p SHLVL'; :)
declare -x SHLVL="2"

Under normal conditions, this would have no visible effect. But because
of the other factors involved, we can see a difference.

[David]

Thanks. Is SHLVL something that is useful to understand?
Is there any documentation about it apart from reading the source?
'man bash' is extremely terse about it. But bash exposes it to users,
so I wonder why, what is it useful for?

[Tim Woodall]

On Fri, 26 Aug 2022, Tim Woodall wrote:

> $ cat Makefile.test
...
>

Many thanks everybody for all of the help understanding this.

Not this exact issue but effectively the same was reported in bug
#944780 and the unbound variable in /etc/bash.bashrc was reported in
#941248

I've now submitted a patch to #944780 that I think fixes all of the
cases where this fails, viz

bash -c 'echo $SHLVL' &
( bash -c 'echo $SHLVL' )
$( bash -c 'echo $SHLVL' )
bash -c 'echo $SHLVL' | cat
coproc bash -c 'echo $SHLVL'

All of which incorrectly (IMO) fail to increment SHLVL

I found this amusing case where I can decrement SHLVL below 1. I haven't
managed to get it to go negative.

$ echo $SHLVL
1
$ cat Makefile.test
all:
echo $$SHLVL

$ make -f Makefile.test | cat

echo $SHLVL

0
$

Tim.