handling of test == by BASH's POSIX mode

[Jon Seymour]

Is there a reason why bash doesn't treat == as an illegal test
operator when running in POSIX mode?

This is problematic because use of test == in scripts that should be
POSIX isn't getting caught when I run them under bash's POSIX mode.
The scripts then fail when run under dash which seems to be stricter
about this.

I have reconfigured my system's default /bin/sh back to /bin/dash to
ensure better POSIX compliance, but it would be nice if I didn't have
to do that.

I am running Ubuntu's distribution of bash, per:

jseymour@ubuntu:~/tracked/git$ uname -a
Linux ubuntu 2.6.32-41-generic #88-Ubuntu SMP Thu Mar 29 13:10:32 UTC
2012 x86_64 GNU/Linux
jseymour@ubuntu:~/tracked/git$ bash --version
GNU bash, version 4.1.5(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

jon.

[Geir Hauge]

2012/5/27 Jon Seymour <jon.s...@gmail.com>:

> Is there a reason why bash doesn't treat == as an illegal test
> operator when running in POSIX mode?

POSIX does not say == is not allowed.

POSIX tells you what the shell should at least be able to do. A POSIX
compliant shell can have whatever other features it likes, as long as
the POSIX features are covered.

> This is problematic because use of test == in scripts that should be
> POSIX isn't getting caught when I run them under bash's POSIX mode.
> The scripts then fail when run under dash which seems to be stricter
> about this.

Don't use non-POSIX features in a POSIX script, and you'll be fine.
http://www.opengroup.org/onlinepubs/9699919799/utilities/contents.html

--
Geir Hauge

[Jon Seymour]

On 27/05/2012, at 17:39, Geir Hauge <geir....@gmail.com> wrote:

> 2012/5/27 Jon Seymour <jon.s...@gmail.com>:
>> Is there a reason why bash doesn't treat == as an illegal test
>> operator when running in POSIX mode?
>
> POSIX does not say == is not allowed.
>
> POSIX tells you what the shell should at least be able to do. A POSIX
> compliant shell can have whatever other features it likes, as long as
> the POSIX features are covered.
>

I guess the question is better phrased thus: what use case is usefully served by having bash's POSIX mode support a superset of test operators than other compliant POSIX shells? As it stands, I can't use bash's POSIX mode to verify the validity or otherwise of a POSIX script because bash won't report these kinds of errors - even when running in POSIX mode.

There is an --enable-strict-posix (?) configuration option. Will this do what I expect?

>
>> This is problematic because use of test == in scripts that should be
>> POSIX isn't getting caught when I run them under bash's POSIX mode.
>> The scripts then fail when run under dash which seems to be stricter
>> about this.
>
> Don't use non-POSIX features in a POSIX script, and you'll be fine.
> http://www.opengroup.org/onlinepubs/9699919799/utilities/contents.html
>

Which is the exactly the point. Practically speaking when I write scripts I expect an interpreter that claims to be running in POSIX mode to give me some help to flag usage of non POSIX idioms. Yes, I can second guess the interpreter by reading the spec, but is this really the most efficient way to catch these kinds of errors?

Jon.

[Dan Douglas]

There are no shells in existence that can do what you want. All major shells
claiming to be POSIX compatible include some superset that can't be disabled.
The only shell I have installed not supporting == in [ is dash, and there are
so many scripts in the wild using == with [ it would be a miracle if your
system didn't break because of it. Even the coreutils /usr/bin/[ supports ==.

Performing that kind of checking, rigorously, in a shell, would be impossible
to do statically anyway. Any such lint tool is limited to lexical analysis
which makes it not very useful for testing unless your script is perfectly
free of side-efffects. And who writes side-effect free shell scripts?

How would the shell check for the correctness of:

"$(rm -rf somepath; echo '[')" x == x ]
--
Dan Douglas

[Andreas Schwab]

Jon Seymour <jon.s...@gmail.com> writes:

> As it stands, I can't use bash's POSIX mode to verify the validity or
> otherwise of a POSIX script because bash won't report these kinds of
> errors - even when running in POSIX mode.

You can't do that anyway: POSIX mode does not disable proper extensions
to POSIX, only those that conflict with POSIX. Specifically, in the
case of the test utility, POSIX makes the behaviour unspecified when a
three argument invocation does not match the POSIX-defined binary
operators.

> There is an --enable-strict-posix (?) configuration option. Will this do
> what I expect?

That just switches the default for POSIX mode.

Andreas.

--
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."

[Jon Seymour]

On Sun, May 27, 2012 at 9:24 PM, Dan Douglas <orm...@gmail.com> wrote:
> On Sunday, May 27, 2012 08:45:46 PM Jon Seymour wrote:
>> On 27/05/2012, at 17:39, Geir Hauge <geir....@gmail.com> wrote:
>>
>> I guess the question is better phrased thus: what use case is usefully
> served by having bash's POSIX mode support a superset of test operators than

> other compliant POSIX shells?  As it stands, I can't use bash's POSIX mode to

> verify the validity or otherwise of a POSIX script because bash won't report
> these kinds of errors - even when running in POSIX mode.
>>
>

> There are no shells in existence that can do what you want. All major shells
> claiming to be POSIX compatible include some superset that can't be disabled.
> The only shell I have installed not supporting == in [ is dash, and there are
> so many scripts in the wild using == with [ it would be a miracle if your
> system didn't break because of it. Even the coreutils /usr/bin/[ supports ==.
>

I take your point that this isn't really a bash problem, but a POSIX
problem - POSIX hasn't provided a way to validate whether a script
only uses features that are required to be supported by POSIX
compliant interpreters. An example of this is the failure to prohibit
support for additional test operators which ultimately results in the
creation of interoperability problems between bash and dash - surely
not a good thing.

However, I think my question still remains unanswered - even if bash
is technically compliant with POSIX, what use case is usefully served
by having bash support a superset of the POSIX test operators while
executing in POSIX mode?

Yes, doing allows bash to execute an ill-defined superset of valid
POSIX scripts, but why is that useful - surely POSIX compliant mode
should only be required to execute strictly compliant POSIX scripts
and if we want to execute a superset, then we should explicitly
specify which superset we want to use, not hope that the POSIX
compliant interpreter installed on a machine happens to support that
superset.

You point about scripts failing when running under dash is a good one.
This is exactly my problem: I replaced /bin/sh -> dash with /bin/sh ->
bash because a 3rd party product installation script failed when dash
was the "POSIX" shell. Which is good - I fixed my installation
problem. Better would have been if the original developer had never
released the non-POSIX script in the first place, something that might
not have happened if the bash POSIX implementation was more
conservative. The fact that I didn't switch back to dash after I
installed the product eventually caused me to inject a non-POSIX test
== operator into some scripts I was working on for the git project -
again, something that would not have happened if bash's POSIX mode was
more conservative.

> Performing that kind of checking, rigorously, in a shell, would be impossible
> to do statically anyway. Any such lint tool is limited to lexical analysis
> which makes it not very useful for testing unless your script is perfectly
> free of side-efffects. And who writes side-effect free shell scripts?
>

> How would the shell check for the correctness of:
>
> "$(rm -rf somepath; echo '[')" x == x ]

I wasn't claiming that static checking would be viable. In fact, the
impossibility of static checking is precisely why it would be useful
to have real POSIX "compliant" interpreters that were as conservative
as possible in the syntax and commands they accepted at least in their
so-called POSIX mode. Currently, for example, if I want to test git sh
scripts, I need to ensure that /bin/sh is pointing at dash, because if
I use bash, errors of this kind can slip through. There really isn't a
more effective way for me to fix address this problem than by
abandoning bash as my POSIX shell and using dash instead since it's
POSIX mode does seem to be more conservative.

Given your example:

jseymour@ubuntu:~/tracked/git$ bash --posix -c '$(rm -rf "/tmp/foo";
echo "[") x == x ] && echo yes'
yes

jseymour@ubuntu:~/tracked/git$ dash -c '$(rm -rf "/tmp/foo"; echo "[")
x == x ] && echo yes'
[: 1: x: unexpected operator

bash --posix doesn't give me any clue that == isn't required to be
supported POSIX interpreters, but at least dash does.

jon.

[Jon Seymour]

On Sun, May 27, 2012 at 9:31 PM, Andreas Schwab <sch...@linux-m68k.org> wrote:
> Jon Seymour <jon.s...@gmail.com> writes:
>
>> As it stands, I can't use bash's POSIX mode to verify the validity or
>> otherwise of a POSIX script because bash won't report these kinds of
>> errors - even when running in POSIX mode.
>

> You can't do that anyway: POSIX mode does not disable proper extensions
> to POSIX, only those that conflict with POSIX.  Specifically, in the
> case of the test utility, POSIX makes the behaviour unspecified when a
> three argument invocation does not match the POSIX-defined binary
> operators.

I understand that the behaviour is unspecitied by POSIX - I didn't
know that before, but I know that now - thanks.

That said, from the point of view of promoting interoperable scripts,
my view is that it (in an ideal world**) would be better if bash chose
to fail, while executing in POSIX mode, in this case.

Yes, it would be annoying to the script writer who has to replace ==
with =, but at least the script eco-system as a whole would benefit
from a more conservative understanding of what it means to be a valid
POSIX script. As it stands now, it is well nigh impossible to avoid
undefined behaviour when the interpreter chooses to gracefully glosses
over the fact that the script is utiising.

** I guess I can except that current bash behaviour is, on balance,
the correct pragmatic decision since there would no doubt be
widespread carnage in the scripting universe if bash was suddenly to
become pickier about how it supports POSIX mode. Is there a case, I
wonder, for enabling more conservative interpretation of test
operators via a shell option of some kind?

>
>> There is an --enable-strict-posix (?) configuration option. Will this do
>> what I expect?
>
> That just switches the default for POSIX mode.

Thanks.

jon.

[Jon Seymour]

On Sun, May 27, 2012 at 11:09 PM, Jon Seymour <jon.s...@gmail.com> wrote:
> On Sun, May 27, 2012 at 9:31 PM, Andreas Schwab <sch...@linux-m68k.org> wrote:

>> Jon Seymour <jon.seymour@gm>

> ** I guess I can except that current bash behaviour is, on balance,

except -> accept

[Dan Douglas]

> POSIX hasn't provided a way to validate whether a script
> only uses features that are required to be supported by POSIX
> compliant interpreters.

I believe that was someone else's point, but yes that would be a problem for
anyone who wanted to implement compliance check warnings.

> even if bash is technically compliant with POSIX, what use case is usefully
> served by having bash support a superset of the POSIX test operators while
executing in POSIX mode?

It's a matter of practicality and the fact that nobody has written it yet. If
you wanted to implement conformance checks without making Bash even bigger and
slower, and harder to maintain, I don't think there would be objections. Bash
just modifies conflicting features to the minimal extent necessary to bring it
into compliance, which seems to be the path of least resistance.

This would be a big job, I think, and not quite at the top of my wish-list.
Right now you can increase the number of things that fail by explicitly
disabling non-POSIX built-ins using the Bash "enable" builtin.

> I wasn't claiming that static checking would be viable. In fact, the
> impossibility of static checking is precisely why it would be useful
> to have real POSIX "compliant" interpreters that were as conservative
> as possible in the syntax and commands they accepted at least in their
> so-called POSIX mode.

Dash is useful for testing. The Bash answer is [[, which CAN do a lot of
special error handling on due to it being a compound command. I wrote a bit
about this here:

http://mywiki.wooledge.org/BashFAQ/031/#Theory

In reality, [[ is one of the very most portable non-POSIX features available.
Most people shouldn't have to worry about avoiding it.

On Sunday, May 27, 2012 11:09:03 PM Jon Seymour wrote:
> That said, from the point of view of promoting interoperable scripts,
> my view is that it (in an ideal world**) would be better if bash chose
> to fail, while executing in POSIX mode, in this case.

In an ideal world, POSIX would define [[, Dash wouldn't exist, and we would
have some resource other than POSIX that specifies what's ACTUALLY portable
between modern shells, so that the only people who have to worry are those
targeting Busybox and Solaris Heirloom, or stubborn curmudgeons who insist on
Dash... some of these things feel like supporting IE6 - it's better to just
not.

> This is exactly my problem: I replaced /bin/sh -> dash with /bin/sh ->
> bash because a 3rd party product installation script failed when dash
> was the "POSIX" shell.

If you need a fast small shell, use mksh. It supports some of the more
essential features of both Bash and Ksh (arrays, ((, [[), and some of its own,
without all the draconian restrictions of dash.

(Note this is querying my package manager, including symbols and source files)

# equery s dash
* app-shells/dash-0.5.7.1
Total files : 73
Total size : 1.14 MiB
# equery s mksh
* app-shells/mksh-9999
Total files : 39
Total size : 1.61 MiB
# equery s bash
* app-shells/bash-4.2_p28
Total files : 464
Total size : 6.33 MiB

--
Dan Douglas

[Jon Seymour]

On Mon, May 28, 2012 at 2:08 AM, Dan Douglas <orm...@gmail.com> wrote:
> ... Bash

> just modifies conflicting features to the minimal extent necessary to bring it
> into compliance, which seems to be the path of least resistance.
>

Sure. I understand that this is a reasonable philosophy given that
aiming for complete avoidance of unspecified behaviour in the POSIX
spec would probably lead to a completely unusable shell.

That said, it is a shame from the point of view of interoperable scripts.

Perhaps I should just accept that dash, being more minimal, does a
better job of being that conformance testing shell?

> This would be a big job, I think, and not quite at the top of my wish-list.
> Right now you can increase the number of things that fail by explicitly
> disabling non-POSIX built-ins using the Bash "enable" builtin.
>

Thanks for that tip.

>
> Dash is useful for testing. The Bash answer is [[, which CAN do a lot of
> special error handling on due to it being a compound command. I wrote a bit
> about this here:
>
> http://mywiki.wooledge.org/BashFAQ/031/#Theory
>
> In reality, [[ is one of the very most portable non-POSIX features available.
> Most people shouldn't have to worry about avoiding it.
>

Thanks, I'll have a read.

jon.

[Eric Blake]

On 05/27/2012 01:15 AM, Jon Seymour wrote:
> Is there a reason why bash doesn't treat == as an illegal test
> operator when running in POSIX mode?

Because POSIX doesn't forbid extensions. Furthermore, POSIX is
considering the standardization of ==:

http://austingroupbugs.net/view.php?id=375

at which point, the use of == would be required to work.

--
Eric Blake ebl...@redhat.com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[Eric Blake]

On 05/27/2012 07:09 AM, Jon Seymour wrote:

> I understand that the behaviour is unspecitied by POSIX - I didn't
> know that before, but I know that now - thanks.
>

> That said, from the point of view of promoting interoperable scripts,
> my view is that it (in an ideal world**) would be better if bash chose
> to fail, while executing in POSIX mode, in this case.

Sounds like you want debian's 'posh', which is even stricter than
'dash', and excellent for pointing out use of extensions by failing
loudly at any encountered extension, including == under test.

[Jon Seymour]

On Tue, May 29, 2012 at 1:08 AM, Eric Blake <ebl...@redhat.com> wrote:
> On 05/27/2012 07:09 AM, Jon Seymour wrote:
>
>> I understand that the behaviour is unspecitied by POSIX - I didn't
>> know that before, but I know that now - thanks.
>>
>> That said, from the point of view of promoting interoperable scripts,
>> my view is that it (in an ideal world**) would be better if bash chose
>> to fail, while executing in POSIX mode, in this case.
>
> Sounds like you want debian's 'posh', which is even stricter than
> 'dash', and excellent for pointing out use of extensions by failing
> loudly at any encountered extension, including == under test.

Thanks, I'll check it out.

jon.