ksh vs sh quoting

[Patrick Welche]

While trying to track down a libtool problem on yesterday's
NetBSD-current/i386, I ended up with:

================= foo ==================
Xsed='sed -e s/^X//'
no_glob_subst='s/\*/\\\*/g'

output_cmd="hello"
echo "output_cmd: ($output_cmd)"
output_cmd="`echo \"X$output_cmd\" | $Xsed -e \"$no_glob_subst\"`"
echo "output_cmd: ($output_cmd)"
========================================
% sh bar (also OK with bash)
output_cmd: (hello)
output_cmd: (hello)
% ksh bar
output_cmd: (hello)
sed: 1: ""s/\*/\\\*/g"": invalid command code "
output_cmd: ()

Any thoughts on what's up with the quoting?

Cheers,

Patrick

[Patrick Welche]

On Tue, Jul 27, 2004 at 04:29:47PM +0100, Patrick Welche wrote:
> Any thoughts on what's up with the quoting?

All explained in TFM.. ksh(1)

The following is a list of things that are affected by the state of the
posix option:
o \" inside double quoted `..` command substitutions: in posix
mode, the \" is interpreted when the command is interpreted; in
non-posix mode, the backslash is stripped before the command
substitution is interpreted. For example, echo "`echo \"hi\"`"
produces `"hi"' in posix mode, `hi' in non-posix mode. To avoid
problems, use the $(...) form of command substitution.

So it seems sh does the non-posix mode, and ksh by default does the posix
mode.. Sure enough set +o posix fixed things for me..

Cheers,

Patrick

[Alan Barrett]

On Tue, 27 Jul 2004, Patrick Welche wrote:
> All explained in TFM.. ksh(1)
>
> The following is a list of things that are affected by the
> state of the posix option:
> o \" inside double quoted `..` command substitutions:
> in posix mode, the \" is interpreted when the command
> is interpreted; in non-posix mode, the backslash is
> stripped before the command substitution is interpreted.
> For example, echo "`echo \"hi\"`" produces `"hi"' in
> posix mode, `hi' in non-posix mode. To avoid problems,
> use the $(...) form of command substitution.
>
> So it seems sh does the non-posix mode, and ksh by default does the
> posix mode.. Sure enough set +o posix fixed things for me..

Backtick command substitution has always been difficult to get right in
complex cases. If I understand the above description correctly, then
non-posix mode is the traditional behaviour, and posix mode is some new
invention.

I recommend that you avoid `...` command substitution, except perhaps in
simple cases. In this case, I recommend changing from

output_cmd="`echo \"X$output_cmd\" | $Xsed -e \"$no_glob_subst\"`"

to

output_cmd="$(echo "X$output_cmd" | $Xsed -e "$no_glob_subst")"

--apb (Alan Barrett)

[Patrick Welche]

The autotools wizards have shown me the error of our ksh's ways, and
pointed me to

http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02__02

The Open Group Base Specifications Issue 6
IEEE Std 1003.1, 2004 Edition

2.2 Quoting

2.2.3 Double-Quotes

Enclosing characters in double-quotes ( "" ) shall preserve the
literal value of all characters within the double-quotes, with the
exception of the characters dollar sign, backquote, and backslash, as
follows:
...
`
The backquote shall retain its special meaning introducing the
other form of command substitution (see Command Substitution ).
The portion of the quoted string from the initial backquote and
the characters up to the next backquote that is not preceded by
a backslash, having escape characters removed, defines that
command whose output replaces "`...`" when the word is
expanded. Either of the following cases produces undefined
results:
+ A single-quoted or double-quoted string that begins, but does
not end, within the "`...`" sequence
+ A "`...`" sequence that begins, but does not end, within the
same double-quoted string

which seems to be directly in opposition to what our ksh(1) man
page says:

The following is a list of things that are affected by the state of the
posix option:
o \" inside double quoted `..` command substitutions: in posix
mode, the \" is interpreted when the command is interpreted; in
non-posix mode, the backslash is stripped before the command
substitution is interpreted. For example, echo "`echo \"hi\"`"
produces `"hi"' in posix mode, `hi' in non-posix mode. To avoid
problems, use the $(...) form of command substitution.

Looking at the standard above, in posix mode, the command executed
should be |echo "hi"|, leading to |hi| being printed, (| used as
quote) not |"hi"|. This is in direct contradition to the man page..
(The shell's behaviour is in accordance with the man page)

Thoughts?

Patrick