BASH quirk with {}

[Kenny McCormack]

Observe:

$ echo /etc/{passwd,group}
/etc/passwd /etc/group
$ echo /etc/{passwd}
/etc/{passwd}
$

csh/tcsh does the right thing in the second case:

% echo /etc/{passwd}
/etc/passwd
%

--
I've been watching cat videos on YouTube. More content and closer to
the truth than anything on Fox.

[Barry Margolin]

In article <mmeci2$fac$2...@news.xmission.com>,

gaz...@shell.xmission.com (Kenny McCormack) wrote:

> Observe:
>
> $ echo /etc/{passwd,group}
> /etc/passwd /etc/group
> $ echo /etc/{passwd}
> /etc/{passwd}
> $
>
> csh/tcsh does the right thing in the second case:
>
> % echo /etc/{passwd}
> /etc/passwd
> %

From the manual:

A correctly-formed brace expansion must contain unquoted opening and
closing braces, and at least one unquoted comma or a valid sequence
expression. Any incorrectly formed brace expansion is left unchanged.

http://www.gnu.org/software/bash/manual/bashref.html#Brace-Expansion

There's no unquoted comma in /etc/{passwd}, so it's not a proper brace
expansion, and it's left unchanged.

It was probably done this way to minimize the incompatibility when brace
expansion was introduced. For instance, it's common to use an unquoted
{} with find -exec, so that shouldn't be treated as a limiting case of
brace expansion.

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

[Janis Papanagnou]

On 24.06.2015 15:49, Kenny McCormack wrote:
> Observe:
>
> $ echo /etc/{passwd,group}
> /etc/passwd /etc/group
> $ echo /etc/{passwd}
> /etc/{passwd}
> $

I had been annoyed/astonished about that in ksh as well.

My guess was that (since you write only literal names there inside the
braces) they thought you could also omit them (probably so that you
can still use braces (but not braces with commas) as part of filenames).

Janis

[Kaz Kylheku]

On 2015-06-24, Kenny McCormack <gaz...@shell.xmission.com> wrote:
> Observe:
>
> $ echo /etc/{passwd,group}
> /etc/passwd /etc/group
> $ echo /etc/{passwd}
> /etc/{passwd}

The behavior of the second example makes it POSIX conforming, though,
because brace expansion is a nonconforming Bash extension.

This is arguably a good thing, since the the more conservatively the syntax
of a nonconforming extension is defined, the less "invasive" it is.

[Kaz Kylheku]

On 2015-06-24, Janis Papanagnou <janis_pa...@hotmail.com> wrote:
> On 24.06.2015 15:49, Kenny McCormack wrote:
>> Observe:
>>
>> $ echo /etc/{passwd,group}
>> /etc/passwd /etc/group
>> $ echo /etc/{passwd}
>> /etc/{passwd}
>> $
>
> I had been annoyed/astonished about that in ksh as well.
>
> My guess was that (since you write only literal names there inside the
> braces) they thought you could also omit them (probably so that you
> can still use braces (but not braces with commas) as part of filenames).

Don't forget the empty {}, which is an important token in the find command's
-exec predicate! Bash's brace expansion likewise leaves that alone.

[Stephane Chazelas]

2015-06-24 16:47:50 +0000, Kaz Kylheku:

By the way. There's a urban legend that says that some versions
of csh where expanding {} to the empty string (implying that
find . -exec cmd {} \; would have to be written with '{}'). I
could never verify that claim.

Does anybody has any verifiable information on that?

--
Stephane

[Kenny McCormack]

In article <mmee9k$fgj$1...@news.m-online.net>,

Janis Papanagnou <janis_pa...@hotmail.com> wrote:
>On 24.06.2015 15:49, Kenny McCormack wrote:
>> Observe:
>>
>> $ echo /etc/{passwd,group}
>> /etc/passwd /etc/group
>> $ echo /etc/{passwd}
>> /etc/{passwd}
>> $
>
>I had been annoyed/astonished about that in ksh as well.

I am glad to hear that I am not alone.
Although, I'm sure it is documented, so it is not a bug.
That's why I called it a "quirk" (which it clearly is).

>My guess was that (since you write only literal names there inside the
>braces) they thought you could also omit them (probably so that you
>can still use braces (but not braces with commas) as part of filenames).

The actual context where this came up is with the following command, which
displays all the open files for all the sshd's running on the system:

# ls -lsa /proc/{$(pidof sshd | sed 's/ /,/g')}/fd

I think you can see that, as written, it works for n > 2. But, alas, the
n = 1 case needs to be special cased.

BTW, the above was typed from memory; I hope I get all the symbols right
and in the right order.

--
There are two kinds of Republicans: Billionaires and suckers.
Republicans: Please check your bank account and decide which one is you.

[Janis Papanagnou]

On 25.06.2015 02:36, Kenny McCormack wrote:
[...]

>
> The actual context where this came up is with the following command, which
> displays all the open files for all the sshd's running on the system:
>
> # ls -lsa /proc/{$(pidof sshd | sed 's/ /,/g')}/fd
>
> I think you can see that, as written, it works for n > 2. But, alas, the
> n = 1 case needs to be special cased.
>
> BTW, the above was typed from memory; I hope I get all the symbols right
> and in the right order.

I'd wonder if that would generally work. Shell expansions seem not to be
performed inside brace expansion in bash or zsh; each of those commands

x=passwd,group ; echo /etc/{$x}
x={passwd,group} ; echo /etc/$x
echo /etc/{$(echo passwd group | sed "s/ /,/g" )}

return just the unexpanded string

/etc/{passwd,group}

while in ksh you get the expected

/etc/passwd /etc/group


Janis

[Barry Margolin]

In article <mmfif5$83g$1...@news.xmission.com>,

gaz...@shell.xmission.com (Kenny McCormack) wrote:

> In article <mmee9k$fgj$1...@news.m-online.net>,
> Janis Papanagnou <janis_pa...@hotmail.com> wrote:
> >On 24.06.2015 15:49, Kenny McCormack wrote:
> >> Observe:
> >>
> >> $ echo /etc/{passwd,group}
> >> /etc/passwd /etc/group
> >> $ echo /etc/{passwd}
> >> /etc/{passwd}
> >> $
> >
> >I had been annoyed/astonished about that in ksh as well.
>
> I am glad to hear that I am not alone.
> Although, I'm sure it is documented, so it is not a bug.
> That's why I called it a "quirk" (which it clearly is).
>
> >My guess was that (since you write only literal names there inside the
> >braces) they thought you could also omit them (probably so that you
> >can still use braces (but not braces with commas) as part of filenames).
>
> The actual context where this came up is with the following command, which
> displays all the open files for all the sshd's running on the system:
>
> # ls -lsa /proc/{$(pidof sshd | sed 's/ /,/g')}/fd
>
> I think you can see that, as written, it works for n > 2. But, alas, the
> n = 1 case needs to be special cased.

I don't think this works for any n, because brace expansion isn't done
after command substitution.

How about doing this:

ls -lsa $(printf "/proc/%s/fd " $(pidof sshd))

> BTW, the above was typed from memory; I hope I get all the symbols right
> and in the right order.

Maybe you actually had an "eval" in it?

eval "ls -lsa /proc/{$(pidof sshd | sed 's/ /,/g')}/fd"

[Barry Margolin]

In article <mmfl0c$9bn$1...@news.m-online.net>,

Janis Papanagnou <janis_pa...@hotmail.com> wrote:

> On 25.06.2015 02:36, Kenny McCormack wrote:
> [...]
> >
> > The actual context where this came up is with the following command, which
> > displays all the open files for all the sshd's running on the system:
> >
> > # ls -lsa /proc/{$(pidof sshd | sed 's/ /,/g')}/fd
> >
> > I think you can see that, as written, it works for n > 2. But, alas, the
> > n = 1 case needs to be special cased.
> >
> > BTW, the above was typed from memory; I hope I get all the symbols right
> > and in the right order.
>
> I'd wonder if that would generally work. Shell expansions seem not to be
> performed inside brace expansion in bash or zsh; each of those commands

Right. Where it becomes really annoying is when you try to use a range.
You can't do

start=10
end=20
echo foo{$start..$end}

This will just echo

foo{10..20}

[Stephane Chazelas]

2015-06-25 00:36:53 +0000, Kenny McCormack:
[...]

> The actual context where this came up is with the following command, which
> displays all the open files for all the sshd's running on the system:
>
> # ls -lsa /proc/{$(pidof sshd | sed 's/ /,/g')}/fd
>
> I think you can see that, as written, it works for n > 2. But, alas, the
> n = 1 case needs to be special cased.
>
> BTW, the above was typed from memory; I hope I get all the symbols right
> and in the right order.

[...]

That wouldn't work in bash or zsh. The , has to be litteral
there for it to work.

~$ b=a,b zsh -c 'echo /{$b,c}/'
/a,b/ /c/
~$ b=a,b ksh -c 'echo /{$b,c}/'
/a/ /b/ /c/
~$ b=a,b mksh -c 'echo /{$b,c}/'
/a/ /b/ /c/
~$ b=a,b bash -c 'echo /{$b,c}/'
/a,b/ /c/

In zsh, you want to use the $^array operator here:

pids=($(pidof sshd))
ls -las /proc/$^pids/fd

Or condensed:

ls -las /proc/${^$(pidof sshd)}/fd

--
Stephane

[Stephane Chazelas]

2015-06-24 21:28:07 -0400, Barry Margolin:
[...]

> Right. Where it becomes really annoying is when you try to use a range.
> You can't do
>
> start=10
> end=20
> echo foo{$start..$end}
>
> This will just echo
>
> foo{10..20}

[...]

It's only in bash that it doesn't work. That {a..b} is initially
a zsh feature (though was extended later by ksh93 and then later
copied with some of the ksh93 extensions by bash)

And {$a..$b} works OK in zsh and ksh93.

{$a,$b} works OK in bash, so the fact that {$a..$b} doesn't work
there looks more like a bug to me.

--
Stephane

[Stephane Chazelas]

2015-06-25 03:20:12 +0200, Janis Papanagnou:
[...]
> x=passwd,group ; echo /etc/{$x}
[...]

> while in ksh you get the expected
>
> /etc/passwd /etc/group

[...]

That's not what I'd expect for that matters.

Interestingly, ksh93 seems to disable split+glob there, but not
mksh:

$ f='p*w*d,i*e' mksh -c 'echo /etc/{$f}'
/etc/passwd /etc/issue
$ f='p*w*d,i*e' ksh93 -c 'echo /etc/{$f}'
/etc/p*w*d /etc/i*e

In mksh, it seems you can escape the comma with quotes, not with
ksh93:

$ f='a,b' mksh -c 'echo /etc/{"$f",passwd}'
/etc/a,b /etc/passwd

$ f='a,b' ksh93 -c 'echo /etc/{"$f",passwd}'
/etc/a /etc/b /etc/passwd

So in mksh, quotes disable the split+glob and coma handling all
at once.

Still in ksh93:

$ ksh93 -c 'echo /etc/{a","b,c}'
/etc/a,b /etc/c

Not much consistency.

--
Stephane

[Janis Papanagnou]

From a language design viewpoint the comma should probably be part of
the syntactical brace expression construct, so not considered in case
it's passed, e.g., through a variable (or other expansions).

What do you think?

Janis

[Stephane Chazelas]

2015-06-25 16:04:47 +0200, Janis Papanagnou:
[...]

> From a language design viewpoint the comma should probably be part of
> the syntactical brace expression construct, so not considered in case
> it's passed, e.g., through a variable (or other expansions).
>
> What do you think?

[...]

That seems to be zsh and bash position. That does make sense to
me. I mostly see it as an interactive-shell feature to save
typing anyway.

You can't really use it to dispatch the content of an array
anyway. mksh is the only one where you can do something
approaching:

$ mksh -c 'echo /{"$1","$2","$3"}/' sh 1 "foo, bar" 3
/1/ /foo, bar/ /3/

That's why you need a different operator for that, like the
$^array of zsh.

That's also what the ^ operator in rc does (and I suspect zsh's
$^array is named after that):

$ rc
; a=('foo' 'bar, baz' blah)
; echo /^$a^/
/foo/ /bar, baz/ /blah/

Though it doesn't work the same for more than one level:

; echo /^$a^/$a
/foo/foo /bar, baz/bar, baz /blah/blah

zsh:
~$ a=('foo' 'bar, baz' blah)
~$ echo /$^a/$^a
/foo/foo /foo/bar, baz /foo/blah /bar, baz/foo /bar, baz/bar, baz /bar, baz/blah /blah/foo /blah/bar, baz /blah/blah

--
Stephane

[Barry Margolin]

In article <20150625131...@chaz.gmail.com>,

The bash documentation says:

Brace expansion is performed before any other expansions, and any
characters special to other expansions are preserved in the result. It
is strictly textual.

So foo{$a,$b}bar works because it expands the brace to foo${a}bar
foo${b}bar, and THEN expands the variables. But this can't work with
"..", because the expansion depends on the values of the variables.

[Stephane Chazelas]

2015-06-25 11:34:52 -0400, Barry Margolin:
[...]

> > And {$a..$b} works OK in zsh and ksh93.
> >
> > {$a,$b} works OK in bash, so the fact that {$a..$b} doesn't work
> > there looks more like a bug to me.
>
> The bash documentation says:
>
> Brace expansion is performed before any other expansions, and any
> characters special to other expansions are preserved in the result. It
> is strictly textual.
>
> So foo{$a,$b}bar works because it expands the brace to foo${a}bar
> foo${b}bar, and THEN expands the variables. But this can't work with
> "..", because the expansion depends on the values of the variables.

[...]

Ah yes, of course, you're right.

Which also explains:

$ a=1 b=2 ac=foo bc=bar bash -c 'echo {$a,$b}c'
foo bar

Where zsh and ksh and csh (where that feature comes from) do

$ a=1 b=2 ac=foo bc=bar zsh -c 'echo {$a,$b}c'
1c 2c
$ a=1 b=2 ac=foo bc=bar ksh -c 'echo {$a,$b}c'
1c 2c
$ a=1 b=2 ac=foo bc=bar csh -c 'echo {$a,$b}c'
1c 2c

--
Stephane

[Martin Vaeth]

Stephane Chazelas <stephane...@gmail.com> wrote:
> Barry Margolin:

>>
>> The bash documentation says:
>>
>> Brace expansion is performed before any other expansions
>

> Which also explains:
>
> $ a=1 b=2 ac=foo bc=bar bash -c 'echo {$a,$b}c'
> foo bar

Yet another horrible misfeature of bash!
So even if you do not use variables in the expansion,
the result is quite the opposite of what one would
expect intuitively!

It is hard to estimate how many scripts are there in
the wild which are malfunctioning or vulnarable like
this bash script

#!/bin/bash
subdir=code/
gcc $subdir{run,debug}/file.c

which does not what you might expect if

subdirrun=/path/to/troyan

is set...

[Barry Margolin]

In article <mmjknr$2gb$1...@speranza.aioe.org>,

It seems like this problem would be noticed immediately when you first
try to use the script. What's the chance that the script would have
variables named subdirrun and subdirdebug (or that there are environment
variables with those names in the process calling the script)?

The fix is trivial:

subdir=code
gcc $subdir/{run,debug}/file.c

[Martin Vaeth]

Not when you have in front of $subdir another path and several
variants of file.c in (optional) directories.

> What's the chance that the script would have
> variables named subdirrun and subdirdebug

Almost zero, and this is exactly the bad thing:
Something which is perhaps optional is replaced by the empty string,
so you easily miss that you use actually a wrong variable name
in the script.

An attacker can give the command a completely unexpected meaning
by setting an environment variables which is *apparently* not
referred to in the script.

> The fix is trivial:

Sure, if you are aware of this bash syntax trap.
How large do you think is the percentage of script writers which
are aware of this trap?

[Kaz Kylheku]

On 2015-06-26, Martin Vaeth <mar...@mvath.de> wrote:
> Stephane Chazelas <stephane...@gmail.com> wrote:
>> Barry Margolin:
>>>
>>> The bash documentation says:
>>>
>>> Brace expansion is performed before any other expansions
>>
>> Which also explains:
>>
>> $ a=1 b=2 ac=foo bc=bar bash -c 'echo {$a,$b}c'
>> foo bar
>
> Yet another horrible misfeature of bash!

Any syntax that doesn't treat identifiers as indivisible units is a misfeature.

If programmers want this, they should be made to work a little harder:
explicitly invoke some sort of re-scanning logic that pastes tokens together.

> It is hard to estimate how many scripts are there in
> the wild which are malfunctioning or vulnarable like
> this bash script

My estimate is: zero. :)

> #!/bin/bash
> subdir=code/
> gcc $subdir{run,debug}/file.c
>
> which does not what you might expect if
>
> subdirrun=/path/to/troyan

This code doesn't ever do what you expect, because it evidently expands to:

gcc $subdirrun/file.c $subdirdebug/file.c

Quite probably, neither variable exists, and so you get
"gcc: fatal error: no input files". If the script is tested at all and
that line is reached, it doesn't work.

Security sensitive scripts have to guard against variable injection
in general. If you can set arbitrary variables before the execution
of the script, you could just do this:

PATH="/path/to/rogue/bin:$PATH"

where in the rogue bin directory we have a rogue gcc executable.

The root of the problem is misplaced trust in the inherited environment.

[Martin Vaeth]

Kaz Kylheku <k...@kylheku.com> wrote:
>> #!/bin/bash
>> subdir=code/
>> gcc $subdir{run,debug}/file.c
>>
>> which does not what you might expect if
>>
>> subdirrun=/path/to/troyan
>
> This code doesn't ever do what you expect

Yes, this is what I wanted to demonstrate.
But the other case causes no harm.

> because it evidently expands to:

I didn't want to make the example overcomplicated,
just demonstrate how it goes against expectations.
As mentioned in another posting, to get code which
might seem to "work" also at a first testing, put another
directory in front of $subdir, and assume that file.c
exists in several directories.

Or just replace gcc by a command which silently ignores
nonexisting files, and assume that the script was not
tested when code/run/file.c or code/debug/file.c existed
(or perhaps it was tested, and somebody realized that the
files were ignored, but the problem this caused was
not so severe that it appeared necessary to hunt for a
bug in an apparently unimportant code path.)

Several other cases are also thinkable; I just wanted to
give the idea how people are tricked by the strange expansion.

> Security sensitive scripts have to guard against variable injection
> in general.

Yes. That's why it is very bad if an unexpected
way of expansion can refer to a variable name which
most people (in particular: average script authors)
are not even aware of.
The behaviour of bash is here not only different than
most other languages but even than any other shell...

> The root of the problem is misplaced trust in the
> inherited environment.

No, the root of the problem is that the expansion breaks
expectations. As already observed here in this group:
People expect even {$a..$b} to work, and they certainly
do not expect that a harmless looking {a,b} can create
new variable names; especially when other shells obey
to the expectations...

[Kaz Kylheku]

On 2015-06-27, Martin Vaeth <mar...@mvath.de> wrote:

> Kaz Kylheku <k...@kylheku.com> wrote:

>> I didn't want to make the example overcomplicated,
>> just demonstrate how it goes against expectations.
>> As mentioned in another posting, to get code which
>> might seem to "work" also at a first testing, put another
>> directory in front of $subdir, and assume that file.c
>> exists in several directories.

Serious scripts use "set -u". Certainly, anything security sensitive should,
in addition to "set -e".

$ set -u
$ echo $TERM{a,b}
bash: TERMa: unbound variable

If that line is executed at all in testing, the problem is flushed out,
unless by some miracle, all of the variables whose names are implied
in the expansion exist.

>> The root of the problem is misplaced trust in the
>> inherited environment.
>
> No, the root of the problem is that the expansion breaks
> expectations. As already observed here in this group:
> People expect even {$a..$b} to work, and they certainly
> do not expect that a harmless looking {a,b} can create
> new variable names; especially when other shells obey
> to the expectations...

I do agree with you that this is a misfeature. More generally, I would say
this:

Programmers don't expect what appear to be separate tokens to *casually*
paste together and be treated as identifiers, in the course of using ordinary
syntax.

Casually means not without going through some explicit hoop involving
explicitly requesting some return to an earlier processing path to have the
justapoxed material tokenized again. For instance "eval" in POSIX syntax,
or the token pasting ## operator in C.

[Martin Vaeth]

Kaz Kylheku <k...@kylheku.com> wrote:
> On 2015-06-27, Martin Vaeth <mar...@mvath.de> wrote:
>
>>> I didn't want to make the example overcomplicated,
>>> just demonstrate how it goes against expectations.
>

> Serious scripts use "set -u".

Scripts in the wild very rarely do this.
For instance, a quick search here with

grep -- -u $(grep -Rl '/bin/.*sh' /usr/bin/*)

gave 0 results (except quite a lot of "false positives").
Moreover, for some scripts, this is also not so easy
(e.g. ./configure or Makefiles which nowadays for
quite a lot of projects assume bash).

> Certainly, anything security sensitive should,
> in addition to "set -e".

I do not agree with this. There are a lot of undesired side
effects, and stopping something is not always what you want.
(Not to speak that you would have to set -e in every subshell, too).
Moreover, -e does not save you from the problem either, if
nonexistent files are tacitly ignored by the command in question.

And another point: Practically *every* script is "security sensitive".
Every ./configure and Makefile might be used to to infiltrate an
undesired codepath (which in the described way can be well hidden),
a lot of scripts are eventually called by root, and also if called
by a user, it might be "security sensitive" enough. (Not to speak
that it might install a Troyan or otherwise compromise later commands).

> Programmers don't expect what appear to be separate tokens to *casually*
> paste together and be treated as identifiers, in the course of using
> ordinary syntax.

This is a good summary.

[Barry Margolin]

In article <mmmaui$tro$1...@speranza.aioe.org>,

Huh? It should report that /file.c doesn't exist and can't be compiled.

>
> > What's the chance that the script would have
> > variables named subdirrun and subdirdebug
>
> Almost zero, and this is exactly the bad thing:
> Something which is perhaps optional is replaced by the empty string,
> so you easily miss that you use actually a wrong variable name
> in the script.

What optional thing are you talking about?

>
> An attacker can give the command a completely unexpected meaning
> by setting an environment variables which is *apparently* not
> referred to in the script.

How is an attacker going to modify the environment of the original
programmer, when he tried out his script?

>
> > The fix is trivial:
>
> Sure, if you are aware of this bash syntax trap.
> How large do you think is the percentage of script writers which
> are aware of this trap?

I'm saying that when they try to run the script the first time they'll
get an error, and discover this trap.

[Barry Margolin]

In article <mmn30p$l9d$1...@speranza.aioe.org>,

Martin Vaeth <mar...@mvath.de> wrote:

> Kaz Kylheku <k...@kylheku.com> wrote:
> >> #!/bin/bash
> >> subdir=code/
> >> gcc $subdir{run,debug}/file.c
> >>
> >> which does not what you might expect if
> >>
> >> subdirrun=/path/to/troyan
> >
> > This code doesn't ever do what you expect
>
> Yes, this is what I wanted to demonstrate.
> But the other case causes no harm.
>
> > because it evidently expands to:
>
> I didn't want to make the example overcomplicated,
> just demonstrate how it goes against expectations.
> As mentioned in another posting, to get code which
> might seem to "work" also at a first testing, put another
> directory in front of $subdir, and assume that file.c
> exists in several directories.
>
> Or just replace gcc by a command which silently ignores
> nonexisting files, and assume that the script was not
> tested when code/run/file.c or code/debug/file.c existed
> (or perhaps it was tested, and somebody realized that the
> files were ignored, but the problem this caused was
> not so severe that it appeared necessary to hunt for a
> bug in an apparently unimportant code path.)

There aren't too many commands that silently ignore nonexisting input
files.

I'm not saying that there's no way that this bug could go unnoticed,
just that it's unlikely.

Nor am I claiming that this behavior of bash is a great feature -- it
just seems to be a consequence of where they happened to shoe-horn brace
expansion into the parser. Although if there weren't already arrays and
indirect variables, you could see how it might occasionally be useful:

foo1=x
foo2=y
foo3=z
cat $foo{1..3}

[Martin Vaeth]

Barry Margolin <bar...@alum.mit.edu> wrote:

> Martin Vaeth <mar...@mvath.de> wrote:
>> >>
>> >> #!/bin/bash
>> >> subdir=code/
>> >> gcc $subdir{run,debug}/file.c
>> >>

>> Not when you have in front of $subdir another path and several
>> variants of file.c in (optional) directories.
>
> Huh? It should report that /file.c doesn't exist and can't be compiled.

subdir=code/
another_path=./
gcc $another_path$subdir{run,debug}/file.c

and you have ./file.c and ./run/file.c and ./debug/file.c

>> Something which is perhaps optional is replaced by the empty string,
>> so you easily miss that you use actually a wrong variable name
>> in the script.
>
> What optional thing are you talking about?

The above could be part of a bigger project, where e.g. only one
the directories ./run and ./debug exists (and e.g. gcc
is replaced by a function ignoring nonexistent files).

> How is an attacker going to modify the environment of the original
> programmer, when he tried out his script?

Not of the programmer but of a person who compiles/runs the
corresponding project. There are many ways of bringing people
to modify environment variables which they do not consider
security relevant: From recommendations in some forums up to
build scripts in disitributions - such harmless looking

> I'm saying that when they try to run the script the first time they'll
> get an error, and discover this trap.

As mentioned above and in another posting: Maybe in the first
simple example not, but it is not too hard to give more realistic
examples ('though they are more lengthy).

[Martin Vaeth]

Barry Margolin <bar...@alum.mit.edu> wrote:
>
> There aren't too many commands that silently ignore nonexisting input
> files.

It used to be the standard behaviour of most commands
(according to unix philosophy "no news is good news").
Meanwhile, many spot at least a message or return nonzero exit status.
Nevertheless, it is not so infrequent to have functions which start
with a loop of the form

for i
do test -e "$i" || test -h "$i" || continue
...

so that you can pass e.g. "*" without caring about nullglob.

Or in a bigger project you might want to have this intentional
to treat optionally some directory trees only if they exist.
For instance, zsh has such type of build system which changes
its behaviour according to the existing files/directories.

> I'm not saying that there's no way that this bug could go unnoticed,
> just that it's unlikely.

For a big damage, it would be sufficient, if only in one project
it goes unnoticed and enters some popular distribution...

> Although if there weren't already arrays and
> indirect variables, you could see how it might occasionally be useful:
>
> foo1=x
> foo2=y
> foo3=z
> cat $foo{1..3}

I am not particularly impressed. It would be better to have the
"expected" behaviour (as in all other shells), and then if you want
the above exceptional handling you can still

eval cat \$foo{1..3}

[Barry Margolin]

In article <mmrb8j$s92$1...@speranza.aioe.org>,

Martin Vaeth <mar...@mvath.de> wrote:

> Barry Margolin <bar...@alum.mit.edu> wrote:
> > Martin Vaeth <mar...@mvath.de> wrote:
> >> >>
> >> >> #!/bin/bash
> >> >> subdir=code/
> >> >> gcc $subdir{run,debug}/file.c
> >> >>
> >> Not when you have in front of $subdir another path and several
> >> variants of file.c in (optional) directories.
> >
> > Huh? It should report that /file.c doesn't exist and can't be compiled.
>
> subdir=code/
> another_path=./
> gcc $another_path$subdir{run,debug}/file.c
>
> and you have ./file.c and ./run/file.c and ./debug/file.c

And if the source files are supposed to be in the subdirectory, why
would there also be files in the current directory? You're combining
multiple artificial, unlikely conditions that are expressly designed to
not be noticed.

>
> >> Something which is perhaps optional is replaced by the empty string,
> >> so you easily miss that you use actually a wrong variable name
> >> in the script.
> >
> > What optional thing are you talking about?
>
> The above could be part of a bigger project, where e.g. only one
> the directories ./run and ./debug exists (and e.g. gcc
> is replaced by a function ignoring nonexistent files).
>
> > How is an attacker going to modify the environment of the original
> > programmer, when he tried out his script?
>
> Not of the programmer but of a person who compiles/runs the
> corresponding project. There are many ways of bringing people
> to modify environment variables which they do not consider
> security relevant: From recommendations in some forums up to
> build scripts in disitributions - such harmless looking

Yes, if the bug gets past initial testing, it could be exploited. But my
contention all along is that it's unlikely to get that far.

>
> > I'm saying that when they try to run the script the first time they'll
> > get an error, and discover this trap.
>
> As mentioned above and in another posting: Maybe in the first
> simple example not, but it is not too hard to give more realistic
> examples ('though they are more lengthy).

I think you're giving LESS realistic examples. It takes a parlay of
several conditions for the bug to go unnoticed.

[Kaz Kylheku]

On 2015-06-29, Martin Vaeth <mar...@mvath.de> wrote:
> Barry Margolin <bar...@alum.mit.edu> wrote:
>> Martin Vaeth <mar...@mvath.de> wrote:
>>> >>
>>> >> #!/bin/bash
>>> >> subdir=code/
>>> >> gcc $subdir{run,debug}/file.c
>>> >>
>>> Not when you have in front of $subdir another path and several
>>> variants of file.c in (optional) directories.
>>
>> Huh? It should report that /file.c doesn't exist and can't be compiled.
>
> subdir=code/
> another_path=./
> gcc $another_path$subdir{run,debug}/file.c
>
> and you have ./file.c and ./run/file.c and ./debug/file.c

... and if you neglect to avail yourself of the incredible advancement in
programming languages known as "set -u" to be informed about accidental
references to nonexistent variable names.

So it serves you right: you're programming in one of the dumbest languages
known to man, and neglecting to follow good practices.

Though the Bash feature is a misfeature, it's basically just piled on top of
all the other POSIX-standard misfeatures. They only don't look like misfeatures
any more because of your decades of being used to them.

Look, above you forgot to quote, and to use use --. Even if we fix the bracing
issue:

$another_path${subdir}{run,debug}/file.c

the lack of quoting means $another_path could expand into "X Y" where X
looks like a command line option to gcc. If you're after bullet-proof,
then you need:

gcc -- "$another_path${subdir}{run,debug}/file.c"

And, guess what? Double quotes suppress Bash's brace expansion. The astute
programmer will then take just that expansion outside of quotes (along with
any literal bits that obviously don't contain space by visual inspection), but
keep the variables quoted:

gcc -- "$another_path$subdir"{run,debug}/file.c

See? Fixing the quoting issue on the variables also eliminates the bracing issue!

[Kaz Kylheku]

On 2015-06-29, Martin Vaeth <mar...@mvath.de> wrote:
> Barry Margolin <bar...@alum.mit.edu> wrote:

>> Although if there weren't already arrays and
>> indirect variables, you could see how it might occasionally be useful:
>>
>> foo1=x
>> foo2=y
>> foo3=z
>> cat $foo{1..3}
>
> I am not particularly impressed. It would be better to have the
> "expected" behaviour (as in all other shells), and then if you want
> the above exceptional handling you can still
>
> eval cat \$foo{1..3}

It occurs to me that shells hould have a lower-cost re-entry point into the
tokenizing, and a properly delimited one in terms of syntax and the scope
of the re-tokenizing.

The problem with eval is that the *whole* command line is subject
to *all* of the expansions before being re-evaluated.

There should be some syntax whereby you can precisely delimit a portion of
the command line that is to be re-scanned:

E.g. suppose, for illustration purposes, that we have $< ... > syntax for
requesting this:

cat $<\$foo{1..3}>

This re-tokenization could take place before field splitting and path and tilde
expansions.

Usually you don't want filesystem material to be embroiled in the syntax.

I.e. $<...> means "do all the $-delimited expansions between the angle
brackets (variables, command substitution, arithmetic, ...) as well
as brace expansions, without doing any path expansions or field splitting, then
substitute the results verbatim.

"do all the $-delimited expansions" implies, of course, that $<...> nests, so
that we can have:

cat $<\$$<$compute_foo>>{1..3}>

$< evaluation nests and so the first step is to process $<$compute_foo>.
compute_foo contains "foo", and so this is replaced by foo. We now
effectively have

cat $<\$foo{1..3}>

and evaluation of the outer $<...> continues, and performs the brace
expansion, resulting in

cat $foo1 $foo2 $foo3

[Martin Vaeth]

Kaz Kylheku <k...@kylheku.com> wrote:
>
> So it serves you right: you're programming in one of the dumbest languages
> known to man, and neglecting to follow good practices.

s/you/almost all projects, thousands of configure scripts etc/

> Look, above you forgot to quote, and to use use --.

Omission of quoting and -- can also be found in thousands of configure
scripts, especially when (as in the example I gave) that it is not
necessary; even more if (as in Makefiles) it is even impossible to
include "tricky" paths into variables names (with $IFS symbols
and others) without causing lots of pain.

> The astute programmer will then

You highly overestimate the skill of programmers who quickly write
a configure to make their C/Java/whatever projects compile. Many are
not even sure concerning details of POSIX (otherwise, they wouldn't
use {a,b} in the first place, since according to POSIX {a,b} should
not expand at all).

[Martin Vaeth]

Kaz Kylheku <k...@kylheku.com> wrote:
>
> There should be some syntax whereby you can precisely delimit a portion of
> the command line that is to be re-scanned

A very primitive form of such a syntax exists (with some limitations).
You can eval "set -- ..." and then use "$@".

> E.g. suppose, for illustration purposes, that we have $< ... > syntax for
> requesting this:
>
> cat $<\$foo{1..3}>

eval set -- \$foo{1..3}
cat "$@"

A limitation of this is that things like && || ; >> etc. are all
treated as ordinary strings. On the other hand, this limitation might
also be considered as a "security feature", because very likely
you would like to consider foo2='&&' in such a context as a filename
and not a shell operator.

[Martin Vaeth]

Barry Margolin <bar...@alum.mit.edu> wrote:
>>
>> and you have ./file.c and ./run/file.c and ./debug/file.c
>
> And if the source files are supposed to be in the subdirectory, why
> would there also be files in the current directory?

This is perhaps done in some projects:
You have the main files in src=. and for those files
which you want to compile with profiling or
debugging CFLAGS you create symlinks in ./run or ./debug.
The advantage of such an approach is that you can simply modify
the directory tree instead of the build system files.
As mentioned, the zsh build system uses something similar
(at least for completion files and help files).

> I think you're giving LESS realistic examples.

Maybe, maybe not. One broken project is one too much.