echo builtin doesn't handle end-of-options flag

[Todd A. Jacobs]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: darwin19.5.0
Compiler: clang
Compilation CFLAGS: -DSSH_SOURCE_BASHRC -Wno-parentheses -Wno-format-security
uname output: Darwin titan.local 19.6.0 Darwin Kernel Version 19.6.0: Thu Jun 18 20:49:00 PDT 2020; root:xnu-6153.141.1~1/RELEASE_X86_64 x86_64
Machine Type: x86_64-apple-darwin19.5.0

Bash Version: 5.0
Patch Level: 18
Release Status: release

Description:
The echo builtin accepts options, but does not seem to handle
`--` correctly as the end of options. The expected behavior
would be for -- to be removed, and any following flag-like
substrings printed as-is.

Repeat-By:
echo "-n" # "" but expecting "-n"
echo -- -n foo # -- -n foo
echo -- "-n foo" # -- -n foo

[Eric Cook]

This is an intentional relic of the past, you really should use printf.

[Eli Schwartz]

On 8/16/20 11:47 AM, Todd A. Jacobs wrote:
> Configuration Information [Automatically generated, do not change]:
> Machine: x86_64
> OS: darwin19.5.0
> Compiler: clang
> Compilation CFLAGS: -DSSH_SOURCE_BASHRC -Wno-parentheses -Wno-format-security
> uname output: Darwin titan.local 19.6.0 Darwin Kernel Version 19.6.0: Thu Jun 18 20:49:00 PDT 2020; root:xnu-6153.141.1~1/RELEASE_X86_64 x86_64
> Machine Type: x86_64-apple-darwin19.5.0
>
> Bash Version: 5.0
> Patch Level: 18
> Release Status: release
>
> Description:
> The echo builtin accepts options, but does not seem to handle
> `--` correctly as the end of options. The expected behavior
> would be for -- to be removed, and any following flag-like
> substrings printed as-is.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html

echo does not accept options, and is not permitted to accept options
including "--".

bash does accept -n, -e, -E in violation of POSIX, unless shopt -s shopt
-s xpg_echo is set, but it doesn't implement -- and I don't really see a
justification to do so. Either enable xpg_echo or use printf.

--
Eli Schwartz
Arch Linux Bug Wrangler and Trusted User

[Eric Blake]

On 8/16/20 10:47 AM, Todd A. Jacobs wrote:

> Description:
> The echo builtin accepts options, but does not seem to handle
> `--` correctly as the end of options.

The correct behavior for echo is described here:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html
"The echo utility shall not recognize the "--" argument in the manner
specified by Guideline 10 of XBD Utility Syntax Guidelines; "--" shall
be recognized as a string operand."

> The expected behavior
> would be for -- to be removed, and any following flag-like
> substrings printed as-is.

Your expectations are wrong, they contradict what POSIX says.

>
> Repeat-By:
> echo "-n" # "" but expecting "-n"

POSIX says this one is implementation-defined; so whether -n is treated
as an option or as a string to echo has to be documented by the
implementation (bash documents treating it as an option).

> echo -- -n foo # -- -n foo

POSIX says this has to output '-- -n foo' and a newline. -n wasn't
first, so you no longer have the implementation-defined behavior, but
well-defined.

> echo -- "-n foo" # -- -n foo

Also well-defined.

I see nothing in your report about bash disobeying POSIX, but rather
confusion on your part about what POSIX actually requires.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org

[Martin Schulte]

Hello Todd,

Eric Cook wrote:
> This is an intentional relic of the past, you really should use printf.

See https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html#tag_20_37_18

BTW/1:

Because of this /bin/echo behaves the same way in most Linux distribution.

BTW/2:

echo -n -n$'\n'

will print a -n in the bash.

Best regards

Martin

[Eli Schwartz]

On 8/16/20 12:21 PM, Eric Blake wrote:
> On 8/16/20 10:47 AM, Todd A. Jacobs wrote:
>
>> Description:
>>          The echo builtin accepts options, but does not seem to handle
>>          `--` correctly as the end of options.
>
> The correct behavior for echo is described here:
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html
> "The echo utility shall not recognize the "--" argument in the manner
> specified by Guideline 10 of XBD Utility Syntax Guidelines; "--" shall
> be recognized as a string operand."
>
>> The expected behavior
>>          would be for -- to be removed, and any following flag-like
>>          substrings printed as-is.
>
> Your expectations are wrong, they contradict what POSIX says.
>
>>
>> Repeat-By:
>>          echo "-n"         # "" but expecting "-n"
>
> POSIX says this one is implementation-defined; so whether -n is treated
> as an option or as a string to echo has to be documented by the
> implementation (bash documents treating it as an option).

Err...

"Implementations shall not support any options."

>>          echo -- -n foo    # -- -n foo
>
> POSIX says this has to output '-- -n foo' and a newline. -n wasn't
> first, so you no longer have the implementation-defined behavior, but
> well-defined.
>
>>          echo -- "-n foo"  # -- -n foo
>
> Also well-defined.
>
> I see nothing in your report about bash disobeying POSIX, but rather
> confusion on your part about what POSIX actually requires.

bash does disobey POSIX, but not in the manner which the report specifies.

[Eric Blake]

On 8/16/20 11:20 AM, Eli Schwartz wrote:

> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html
>
> echo does not accept options, and is not permitted to accept options
> including "--".
>
> bash does accept -n, -e, -E in violation of POSIX, unless shopt -s shopt
> -s xpg_echo is set, but it doesn't implement -- and I don't really see a
> justification to do so.

Not quite: https://www.austingroupbugs.net/view.php?id=1222 says that
accepting -e and -E will become acceptable in the next revision of POSIX
(accepting -n was already acceptable on non-XSI systems).

> Either enable xpg_echo or use printf.

POSIX recommends using printf. My personal recommendation is to avoid
xpg_echo except when trying to break things - there are a number of what
claim to be bash scripts that are not aware of the effects of xpg_echo
and which operate incorrectly when you actually set it; whereas using
printf is reliably portable.

[Todd A. Jacobs]

On Aug 16, 2020, at 12:21 PM, Eric Blake <ebl...@redhat.com> wrote:

> I see nothing in your report about bash disobeying POSIX, but rather confusion on your part about what POSIX actually requires.

I didn't say it violated POSIX. My expectations were set by the Bash Manual, which describes the behavior I'm expecting in the Shell Builtin Commands section:

> Unless otherwise noted, each builtin command documented as accepting options preceded by ‘-’ accepts ‘--’ to signify the end of the options.

To be fair, the documentation later calls out that "echo does not interpret -- to mean the end of options", but this wasn't obvious on first reading. If the demonstrated behavior is correct and/or desirable, then it's not a bug. I still found it surprising, but I'm honestly not sure how to improve the documentation to draw more attention to the known behavior. Maybe it could be bolded, or listed as a caveat, because it's easy to overlook.

[Chet Ramey]

On 8/16/20 12:20 PM, Eli Schwartz wrote:
>
>> Bash Version: 5.0
>> Patch Level: 18
>> Release Status: release
>>

> bash does accept -n, -e, -E in violation of POSIX, unless shopt -s shopt
> -s xpg_echo is set,

That's not a POSIX violation.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU ch...@case.edu http://tiswww.cwru.edu/~chet/

[Chet Ramey]

On 8/16/20 12:21 PM, Eric Blake wrote:

> POSIX says this one is implementation-defined; so whether -n is treated as
> an option or as a string to echo has to be documented by the implementation
> (bash documents treating it as an option).

It's not technically an option; it's a first operand that is treated
specially.

[Chet Ramey]

On 8/16/20 12:26 PM, Eli Schwartz wrote:

> Err...
>
> "Implementations shall not support any options."

Yes, that's why the -n/-E/-e aren't described as options -- that would
turn on the special `--' processing -- but as first operands that enable
special treatment.