Problem with ksh test operator

[Jeffery Small]

I am running Solaris 10 on a Sun SPARC system.

I have a problem testing strings for equality within a Korn shell script
and am hoping someone knows the cause of the problem and a simple fix.

Consider two directories. One is empty and the other contains the following
three files:

PICT_01.jpg
PICT_02.jpg
PICT_03.jpg

Here is a short ksh script:

#! /bin/ksh
LIST="PICT*"
echo ${LIST}
if [[ "${LIST}" = 'PICT*' ]] ; then
echo "No files found."
else
echo "OK. Process files ..."
fi

When making the assignment LIST="PICT*", this will return 'PICT*' in the
empty directory and 'PICT_01.jpg PICT_02.jpg PICT_03.jpg' in the directory
with the three files. This is confirmed by echo on the third line of
the script. The subsequent test attempts to trap the condition where no
matching files are found. However, this fails, and in both directories
the script reports "No files found."

The reason for this is that the ksh [[ operator treats the second string
as a pattern and not a literal string, apparently even when enclosed in
single quotes. From the ksh(1) man page under Conditional Expressions:

string = pattern True, if string matches pattern.

I have tried to backslash escape the "*" in the pattern without success.
If you change the line to:

if [[ "${LIST}" = 'PICT\*' ]] ; then

then the test always fails in both directories. Apparently the backslash
is being taken as a literal rather than an escape character. I have also
tried every combination of single and double quotes, with nothing working.
If there is a trick to get the [[ operator to compare two literal strings
rather than a pattern, I would really appreciate hearing about it.

So I read the test(1) manual page and it reports the following:

string1 = string2 True if the strings string1 and
string2 are identical.

Yes, this is what I want! So I change:

if [[ "${LIST}" = 'PICT*' ]] ; then
to
if [ "${LIST}" = 'PICT*' ] ; then

but this changes nothing. OK, maybe '[' is treated internal to ksh. So
I then tried #!/bin/sh (no good) and then change the line to:

if /bin/test "${LIST}" = 'PICT*' ; then

Nope. How about:

if $(/bin/test "${LIST}" = 'PICT*') ; then

Still no success. So what's the trick? Do I have to write my own external
string comparison routine in some other language such as perl(1)?

This seems like such a no-brainer, but I have lost considerable time going
down dead ends here. Doesn't this seem like a bug in the ksh handling of
the test operator -- especially when calling the external /bin/test?

Any insights would be appreciated. Thanks.
--
Jeffery Small

[Barry Margolin]

In article <M99nD...@cjsa.com>, je...@cjsa.com (Jeffery Small) wrote:

> I am running Solaris 10 on a Sun SPARC system.
>
> I have a problem testing strings for equality within a Korn shell script
> and am hoping someone knows the cause of the problem and a simple fix.
>
> Consider two directories. One is empty and the other contains the following
> three files:
>
> PICT_01.jpg
> PICT_02.jpg
> PICT_03.jpg
>
> Here is a short ksh script:
>
> #! /bin/ksh
> LIST="PICT*"
> echo ${LIST}
> if [[ "${LIST}" = 'PICT*' ]] ; then
> echo "No files found."
> else
> echo "OK. Process files ..."
> fi
>
> When making the assignment LIST="PICT*", this will return 'PICT*' in the
> empty directory and 'PICT_01.jpg PICT_02.jpg PICT_03.jpg' in the directory
> with the three files. This is confirmed by echo on the third line of

No, this doesn't happen during the assignment. The wildcard expansion
happens on the echo line, because you didn't quote the variable. Two
transformations are done after unquoted variable expansion: wildcards
and word splitting.

If you do:

echo "${LIST}"

it should print 'PICT*' in both directories, proving that the assignment
was the same.

Try:

LIST=$(echo PICT*)

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

[Jeffery Small]

I wrote:

> I am running Solaris 10 on a Sun SPARC system.
>
> I have a problem testing strings for equality within a Korn shell script
> and am hoping someone knows the cause of the problem and a simple fix.
>
> Consider two directories. One is empty and the other contains the following
> three files:
>
> PICT_01.jpg
> PICT_02.jpg
> PICT_03.jpg
>
> Here is a short ksh script:
>
> #! /bin/ksh
> LIST="PICT*"
> echo ${LIST}
> if [[ "${LIST}" = 'PICT*' ]] ; then
> echo "No files found."
> else
> echo "OK. Process files ..."
> fi
>
> When making the assignment LIST="PICT*", this will return 'PICT*' in the
> empty directory and 'PICT_01.jpg PICT_02.jpg PICT_03.jpg' in the directory
> with the three files. This is confirmed by echo on the third line of

And Barry Margolin <bar...@alum.mit.edu> responded:

> No, this doesn't happen during the assignment. The wildcard expansion
> happens on the echo line, because you didn't quote the variable. Two
> transformations are done after unquoted variable expansion: wildcards
> and word splitting.
>
> If you do:
>
> echo "${LIST}"
>
> it should print 'PICT*' in both directories, proving that the assignment
> was the same.
>
> Try:
>
> LIST=$(echo PICT*)

Barry:

Thank you for the great information.

So filename expansion does not occur during variable assignment! After
30 years of UNIX programming, this is news to me. This explains a lot of
past confusion I have had with scripts. Where is this documented? I just
searched through my Kornshell book and the ksh(1) manual page and found no
discussion of this. It was always my understanding that expansion took
place when either freely expressed or within double quotes ("..."), and was
suppressed within single quotes ('...') and when the special character(s)
were escaped. What are the precise rules for expansion? I'm still finding
inconsistencies. For example, although tilde also modifies filename paths,
tilde expansion does not work the same way file expansion does.

In my directory with the three PICT* files, I just tried the following:

X="PICT*"
echo "$X" produces: PICT*
echo '$X' produces: $X
echo $X produces: PICT_01.jpg PICT_02.jpg PICT_03.jpg

X=PICT*
echo "$X" produces: PICT*
echo '$X' produces: $X
echo $X produces: PICT_01.jpg PICT_02.jpg PICT_03.jpg

X="~"
echo "$X" produces: ~
echo '$X' produces: $X
echo $X produces: ~

X=~
echo "$X" produces: /home/jeff
echo '$X' produces: $X
echo $X produces: /home/jeff

I then created three files in $HOME: PICT_AA.jpg PICT_BB.jpg PICT_CC.jpg

# echo ~/PICT*
/u/jeff/PICT_AA.jpg /u/jeff/PICT_BB.jpg /u/jeff/PICT_CC.jpg

OK, let's try this:

X="~/PICT*"
echo "$X" produces: ~/PICT*
echo '$X' produces: $X
echo $X produces: ~/PICT*

X=~/PICT*
echo "$X" produces: /home/jeff/PICT*
echo '$X' produces: $X
echo $X produces: /home/jeff/PICT_AA.jpg \
/home/jeff/PICT_BB.jpg \
/home/jeff/PICT_CC.jpg

During variable assignment unquoted filename characters are not expanded,
but the tilde character is expanded, unless there is a double-quoted string
with both a tilde and a filename wildcard character, in which then neither
is expanded. Is there a logic to all of this?


I also want to thank David B. Chorlian who wrote directly to me and suggested
replacing the test within my original script with the following:

Why not
if ls | grep -s PICT > /dev/null

echo "OK. Process files ..."

else
echo "No files found."
fi

which is a similar solution to the one offered above by Barry. I
appreciate both responses.

Regards,
--
Jeffery Small

[Barry Margolin]

In article <M9C2J...@cjsa.com>, je...@cjsa.com (Jeffery Small) wrote:

> During variable assignment unquoted filename characters are not expanded,
> but the tilde character is expanded, unless there is a double-quoted string
> with both a tilde and a filename wildcard character, in which then neither
> is expanded. Is there a logic to all of this?

Variables often hold pathnames, it's common to want to use tilde there.
But good programming practice is to quote these variables when you use
them (in case the filename contains whitespace or wildcards), so the
tilde has to be expanded during the assignment.

And for the sake of variables like PATH, which contain a list of
colon-separated pathnames, tilde is expanded either at the beginning of
the value or immediately following ':'.

I'm not sure why wildcards aren't expanded during assignment, I guess
it's just a decision Bourne made, and we're stuck with it for
compatibility.

[Jeffery Small]

Barry Margolin <bar...@alum.mit.edu> writes:

>In article <M9C2J...@cjsa.com>, je...@cjsa.com (Jeffery Small) wrote:

>> During variable assignment unquoted filename characters are not expanded,
>> but the tilde character is expanded, unless there is a double-quoted string
>> with both a tilde and a filename wildcard character, in which then neither
>> is expanded. Is there a logic to all of this?

>Variables often hold pathnames, it's common to want to use tilde there.
>But good programming practice is to quote these variables when you use
>them (in case the filename contains whitespace or wildcards), so the
>tilde has to be expanded during the assignment.

>And for the sake of variables like PATH, which contain a list of
>colon-separated pathnames, tilde is expanded either at the beginning of
>the value or immediately following ':'.

>I'm not sure why wildcards aren't expanded during assignment, I guess
>it's just a decision Bourne made, and we're stuck with it for
>compatibility.

Thanks for all ther information, Barry.

Regards,
--
Jeff

[David Combs]

1: ALL of the ksh variable-expansion stuff discussed in
this thread; what if any parts of it do NOT work
the same in sh (Bourne)?

2: The order in which things get expanded: I recall seeing
a discussion of that somewhere, the rules laid out in
detail. But I forget where I saw it.

Any recollection of where that might be?


Thanks!

David

[Jeffery Small]

dkc...@panix.com (David Combs) writes:

>1: ALL of the ksh variable-expansion stuff discussed in
> this thread; what if any parts of it do NOT work
> the same in sh (Bourne)?

David:

I haven't done any test between sh(1) and ksh(1), but I assume that they
work similarly regarding these types of expansions. My confusion was not
realizing that filename expansion did not occur during initial assignment.
If there are differences, I would also like to know what they are.

>2: The order in which things get expanded: I recall seeing
> a discussion of that somewhere, the rules laid out in
> detail. But I forget where I saw it.
>
> Any recollection of where that might be?

No, a comprehensive source for this information was never mentioned, but
I would love to see this too. I did not find it in the manual pages or in
my Korn Shell book. Does anyone else reading this thread know of a place
online, or a book on shell programming that discusses this?

Regards,
--
Jeff

[Chris Davies]

Jeffery Small <je...@cjsa.com> wrote:
> Why not
> if ls | grep -s PICT > /dev/null

You can simply that further by testing the exit status of ls directly
(no need for the grep):

if ls -d PICT* >/dev/null 2>&1
then
...

Chris

[Barry Margolin]

In article <M9stq...@cjsa.com>, je...@cjsa.com (Jeffery Small) wrote:

> dkc...@panix.com (David Combs) writes:
> >2: The order in which things get expanded: I recall seeing
> > a discussion of that somewhere, the rules laid out in
> > detail. But I forget where I saw it.
> >
> > Any recollection of where that might be?
>
> No, a comprehensive source for this information was never mentioned, but
> I would love to see this too. I did not find it in the manual pages or in
> my Korn Shell book. Does anyone else reading this thread know of a place
> online, or a book on shell programming that discusses this?

I haven't gone through it enough to tell how clear it is, but I expect
the POSIX Shell specification describes it:

http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html

[Jeffery Small]

Barry Margolin <bar...@alum.mit.edu> writes:

>In article <M9stq...@cjsa.com>, je...@cjsa.com (Jeffery Small) wrote:

>> dkc...@panix.com (David Combs) writes:
>> >2: The order in which things get expanded: I recall seeing
>> > a discussion of that somewhere, the rules laid out in
>> > detail. But I forget where I saw it.
>> >
>> > Any recollection of where that might be?
>>
>> No, a comprehensive source for this information was never mentioned, but
>> I would love to see this too. I did not find it in the manual pages or in
>> my Korn Shell book. Does anyone else reading this thread know of a place
>> online, or a book on shell programming that discusses this?

>I haven't gone through it enough to tell how clear it is, but I expect
>the POSIX Shell specification describes it:

>http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html

Barry, thanks for the link. I'll take a closer look at this.

Regards,
--
Jeff

[David Combs]

It *may* have been something by Kernighan. I'll have to go look.

David