rmdir('/unix_path/to/dir') not working with 5.18.1?

[John E. Malmberg]

I am trying to get the GNU make 4.0.90 self tests run on VMS, and they
are written in perl.

I am making some progress, but have found some issues.

I can not seem to do a rmdir() of an absolute or relative Unix path with
Perl 5.18.1.

I have worked around the issue by replacing the rmdir() builtin method
with one that translates the filename to VMS format and then calls
unlink().

LION> perl --version

This is perl 5, version 18, subversion 1 (v5.18.1) built for VMS_IA64

LION> show log decc$*

(LNM$PROCESS_TABLE)

"DECC$EFS_CASE_PRESERVE" = "ENABLE"
"DECC$EFS_CHARSET" = "ENABLE"
"DECC$FILENAME_UNIX_NOVERSION" = "ENABLE"
"DECC$FILENAME_UNIX_REPORT" = "ENABLE"
"DECC$READDIR_DROPDOTNOTYPE" = "ENABLE"

LION> show log perl_*

(LNM$PROCESS_TABLE)

"PERL_ROOT" = "LION$DKA0:[SYS0.SYSCOMMON.perl-5_18.]"


I also noticed that opendir('/search_list/directory') fails when
'directory' is only in the first member of the search list. I am not
sure what the issue there is. I think the solution for that will be
more complex.

I have worked around that issue by decoding the search list and
recreating the path to used for opendir.

Regards,
-John
Personal Opinion Only

[Craig A. Berry]

On Feb 24, 2014, at 11:33 PM, John E. Malmberg <malm...@Encompasserve.org> wrote:


> I can not seem to do a rmdir() of an absolute or relative Unix path with Perl 5.18.1.

It works unless DECC$FILENAME_UNIX_REPORT is defined. That's not particularly well tested and you definitely found a bug. If that's defined, when rmdir calls the internal stat routine, which calls fileify, then 'abc/xyz' becomes 'abc/xyz.DIR;1'. Then stat converts it to VMS format and we get something like 'D0:[craig.TEST.abc]xyz.DIR^;1'.

So we probably need to have fileify omit the version with unix report enabled.

> I have worked around the issue by replacing the rmdir() builtin method with one that translates the filename to VMS format and then calls unlink().
>
> LION> perl --version
>
> This is perl 5, version 18, subversion 1 (v5.18.1) built for VMS_IA64
>
> LION> show log decc$*
>
> (LNM$PROCESS_TABLE)
>
> "DECC$EFS_CASE_PRESERVE" = "ENABLE"
> "DECC$EFS_CHARSET" = "ENABLE"

For future reference, those two are on by default in Perl 5.18.x.

> "DECC$FILENAME_UNIX_NOVERSION" = "ENABLE"
> "DECC$FILENAME_UNIX_REPORT" = "ENABLE"
> "DECC$READDIR_DROPDOTNOTYPE" = "ENABLE"
>
> LION> show log perl_*
>
> (LNM$PROCESS_TABLE)
>
> "PERL_ROOT" = "LION$DKA0:[SYS0.SYSCOMMON.perl-5_18.]"
>
>
> I also noticed that opendir('/search_list/directory') fails when 'directory' is only in the first member of the search list. I am not sure what the issue there is. I think the solution for that will be more complex.

Not sure what's going on there but I'll try to have a look soonish.

> I have worked around that issue by decoding the search list and recreating the path to used for opendir.
>

________________________________________
Craig A. Berry
mailto:craig...@mac.com

"... getting out of a sonnet is much more
difficult than getting in."
Brad Leithauser

[John E. Malmberg]

On 2/25/2014 7:02 PM, Craig A. Berry wrote:
>
> On Feb 24, 2014, at 11:33 PM, John E. Malmberg <malm...@Encompasserve.org> wrote:

>> I can not seem to do a rmdir() of an absolute or relative Unix
>> path with Perl 5.18.1.
>
> It works unless DECC$FILENAME_UNIX_REPORT is defined. That's not
> particularly well tested and you definitely found a bug. If that's
> defined, when rmdir calls the internal stat routine, which calls
> fileify, then 'abc/xyz' becomes 'abc/xyz.DIR;1'.

If fileify is converting 'abc/xyz' to 'abc/xyz.DIR;1', that is a bug
that will break a lot of stuff. With the DECC$EFS_CHARSET enabled, that
should never happen anywhere.

Not only should the ;1 not be present, the ".DIR" should not be present
either.

The ".DIR" should only be present with DECC$EFS_CHARSET disabled because
that is what the older conversions expected, and on ODS-2 it could be
removed on a reverse conversion.

> Then stat converts it to VMS format and we get something like
> 'D0:[craig.TEST.abc]xyz.DIR^;1'.

abc/xyz.dir would be converted to [.abc]xyz^.dir or [.abc]xyz^.dir.dir.

> So we probably need to have fileify omit the version with unix
> report enabled.

And the .DIR in UNIX paths, unless is
foo/bar.dir <=> [.foo]bar^.dir.dir.

Thanks,
-John

[Craig A. Berry]

On Feb 26, 2014, at 11:35 PM, John E. Malmberg <malm...@Encompasserve.org> wrote:

> On 2/25/2014 7:02 PM, Craig A. Berry wrote:
>>
>> On Feb 24, 2014, at 11:33 PM, John E. Malmberg <malm...@Encompasserve.org> wrote:
>
>>> I can not seem to do a rmdir() of an absolute or relative Unix
>>> path with Perl 5.18.1.
>>
>> It works unless DECC$FILENAME_UNIX_REPORT is defined. That's not
>> particularly well tested and you definitely found a bug. If that's
>> defined, when rmdir calls the internal stat routine, which calls
>> fileify, then 'abc/xyz' becomes 'abc/xyz.DIR;1'.
>
> If fileify is converting 'abc/xyz' to 'abc/xyz.DIR;1', that is a bug that will break a lot of stuff. With the DECC$EFS_CHARSET enabled, that should never happen anywhere.
>
> Not only should the ;1 not be present, the ".DIR" should not be present either.

I think we've had this discussion before. Fileify really has to append the .DIR;1 because there are too many things, rmdir being one of them, that simply cannot operate on a directory spec but only on a directory file. That's really the point of fileify and we wouldn't call it if we didn't have to.

>> Then stat converts it to VMS format and we get something like
> > 'D0:[craig.TEST.abc]xyz.DIR^;1'.

I tested this with decc$to_vms using the example in on-line help and it does not escape a semicolon that is part of a valid version spec. It does escape it if the version spec is not valid (or if the semicolon appears somewhere else in the name):

$ mcr []to_vms abc/xyz.dat;32767
Translating: abc/xyz.dat;32767
file: [.ABC]XYZ.DAT;32767
1 files found
$ mcr []to_vms abc/xyz.dat;123456789
Translating: abc/xyz.dat;123456789
file: [.ABC]XYZ.DAT^;123456789
1 files found

Neither DECC$EFS_CHARSET nor DECC$FILENAME_UNIX_REPORT has any impact on this. So I've changed vmsify in Perl to do exactly the same thing the CRTL does. Well, not exactly because the CRTL honors DECC$FILENAME_UNIX_NO_VERSION by escaping the semicolon even if the version spec is valid:

$ DEFINE DECC$FILENAME_UNIX_NO_VERSION 1
$ mcr []to_vms abc/xyz.dat;32767
Translating: abc/xyz.dat;32767
file: [.ABC]XYZ.DAT^;32767
1 files found

I haven't (yet) made Perl do this. It wouldn't be hard to do, but I'm a little skeptical about trying to support the whole wilderness of feature logicals.

[Craig A. Berry]

On Feb 24, 2014, at 11:33 PM, John E. Malmberg <malm...@Encompasserve.org> wrote:

> I also noticed that opendir('/search_list/directory') fails when 'directory' is only in the first member of the search list. I am not sure what the issue there is. I think the solution for that will be more complex.

I have not been able to reproduce this. If you post a reproducer I'll look into it.

[John E. Malmberg]

On 3/7/2014 8:18 AM, Craig A. Berry wrote:
>
> On Feb 26, 2014, at 11:35 PM, John E. Malmberg
> <malm...@Encompasserve.org> wrote:
>
>> On 2/25/2014 7:02 PM, Craig A. Berry wrote:
>>>
>>> On Feb 24, 2014, at 11:33 PM, John E. Malmberg
>>> <malm...@Encompasserve.org> wrote:
>>
>>>> I can not seem to do a rmdir() of an absolute or relative Unix
>>>> path with Perl 5.18.1.
>>>
>>> It works unless DECC$FILENAME_UNIX_REPORT is defined. That's not
>>> particularly well tested and you definitely found a bug. If
>>> that's defined, when rmdir calls the internal stat routine, which
>>> calls fileify, then 'abc/xyz' becomes 'abc/xyz.DIR;1'.
>>
>> If fileify is converting 'abc/xyz' to 'abc/xyz.DIR;1', that is a
>> bug that will break a lot of stuff. With the DECC$EFS_CHARSET
>> enabled, that should never happen anywhere.
>>
>> Not only should the ;1 not be present, the ".DIR" should not be
>> present either.
>
> I think we've had this discussion before. Fileify really has to
> append the .DIR;1 because there are too many things, rmdir being one
> of them, that simply cannot operate on a directory spec but only on a
> directory file. That's really the point of fileify and we wouldn't
> call it if we didn't have to

In a UNIX format filespecification, the .DIR or .DIR;1 is always a bug.
Older CRTLs did that, but now you have to enable a DECC feature to get
that mis-behaivor.

With DECC$EFS_CHARSET disabled, this bug can be detected and compensated
for, and since existing code expects the bug, for DECC$EFS_CHARSET
disabled, fileify should continue to put the .DIR on for now.

When I last updated fileify, I updated the documentation about this and
that programs should not be written to depend on the .DIR being added
because it is a bug, and they should not depend on the bug being present
in future versions of Perl.

With DECC$EFS_CHARSET enabled, fileify must not add a .DIR to a UNIX
file path. If you do that, you can not have /file.dir/xyz as a legal
file specification, and that shows up quite a bit.

It seems that in build procedures foo/.dir/bar shows up a lot, so this
bug is a big deal.

It is one of the reason that I could not make EFS character set
processing enabled by default. The ODS-2 to UNIX conversions were doing
several things wrong, and programs were expecting that.

But proper support of EFS character set prohibited doing the conversions
wrong, so the perl self tests needed to know which mode to expect.

>>> Then stat converts it to VMS format and we get something like
>>> 'D0:[craig.TEST.abc]xyz.DIR^;1'.
>
> I tested this with decc$to_vms using the example in on-line help and
> it does not escape a semicolon that is part of a valid version spec.
> It does escape it if the version spec is not valid (or if the
> semicolon appears somewhere else in the name):

> $ mcr []to_vms abc/xyz.dat;32767 Translating: abc/xyz.dat;32767 file:
> [.ABC]XYZ.DAT;32767 1 files found $ mcr []to_vms
> abc/xyz.dat;123456789 Translating: abc/xyz.dat;123456789 file:
> [.ABC]XYZ.DAT^;123456789 1 files found

The semicolon version delimiter is actually allowed by one of the
Unix/Posix/Linux specifications that I read, but is rare in Unix because
the shell uses it to terminate the command, and the file systems usually
just treat it as part of the filename, not as a version.

So it is not illegal for decc$to_vms to treat it as a version, but it is
probably not the behavior that a ported program would expect.

> Neither DECC$EFS_CHARSET nor DECC$FILENAME_UNIX_REPORT has any impact
> on this. So I've changed vmsify in Perl to do exactly the same thing
> the CRTL does. Well, not exactly because the CRTL honors
> DECC$FILENAME_UNIX_NO_VERSION by escaping the semicolon even if the
> version spec is valid:

Then Perl should also be using that feature.

> $ DEFINE DECC$FILENAME_UNIX_NO_VERSION 1
> $ mcr []to_vms
> abc/xyz.dat;32767 Translating: abc/xyz.dat;32767 file:
> [.ABC]XYZ.DAT^;32767 1 files found
>
> I haven't (yet) made Perl do this. It wouldn't be hard to do, but
> I'm a little skeptical about trying to support the whole wilderness
> of feature logicals.

It is something that IMHO is critical to add, and I encountered this in
the test harness for GNU make and had to modify the code to work around it.

There is almost no reason that most users of UNIX format file
specifications would want DECC$FILENAME_UNIX_NO_VERSION enabled any time
that DECC$FILENAME_REPORT_UNIX is enabled.

And now that I have written this, my memory is coming back to me as to
why I coded the conversion routines to always escape what looked like
versions on UNIX.

The DECC$FILENAME_UNIX_NO_VERSION used to only affect the readdir()
function and no other conversions. So that mean that you could do a
readdir() of [foo]abc^.bar.123 with report Unix and get "abc.bar.123" as
a filename, which when combined with the original path, resulted in
foo/abc.bar.123.

But because decc$to_vms was not honoring the
DECC$FILENAME_UNIX_NO_VERSION settings in the older VMS versions, the
resulting foo/abc.bar.123 could not be used as a path, unless we
converted it back to the original VMS format path.

Having the vmsify put the escape on extra dots or version delimiters
allowed these file specifications to work in Perl. I just did not key
the behavior off of the "no_version" feature.

At the time, I did not know that there was a UNIXY standard that allowed
the ; in file specifications to support versions. I do not have that
URL handy though.

Regards,
-John

[Craig A. Berry]

On Mar 7, 2014, at 6:49 PM, John E. Malmberg <malm...@Encompasserve.org> wrote:

> On 3/7/2014 8:18 AM, Craig A. Berry wrote:
>>
>> On Feb 26, 2014, at 11:35 PM, John E. Malmberg
>> <malm...@Encompasserve.org> wrote:
>>
>>> On 2/25/2014 7:02 PM, Craig A. Berry wrote:
>>>>
>>>> On Feb 24, 2014, at 11:33 PM, John E. Malmberg
>>>> <malm...@Encompasserve.org> wrote:
>>>
>>>>> I can not seem to do a rmdir() of an absolute or relative Unix
>>>>> path with Perl 5.18.1.
>>>>
>>>> It works unless DECC$FILENAME_UNIX_REPORT is defined. That's not
>>>> particularly well tested and you definitely found a bug. If
>>>> that's defined, when rmdir calls the internal stat routine, which
>>>> calls fileify, then 'abc/xyz' becomes 'abc/xyz.DIR;1'.
>>>
>>> If fileify is converting 'abc/xyz' to 'abc/xyz.DIR;1', that is a
>>> bug that will break a lot of stuff. With the DECC$EFS_CHARSET
>>> enabled, that should never happen anywhere.
>>>
>>> Not only should the ;1 not be present, the ".DIR" should not be
>>> present either.
>>
>> I think we've had this discussion before. Fileify really has to
>> append the .DIR;1 because there are too many things, rmdir being one
>> of them, that simply cannot operate on a directory spec but only on a
>> directory file. That's really the point of fileify and we wouldn't
>> call it if we didn't have to
>
> In a UNIX format filespecification, the .DIR or .DIR;1 is always a bug.

I suppose that's true if the user asked for Unix report mode and if we were reporting filespecs to the user. None of that has anything to do with the purpose of fileify, which is to provide the VMS-native filename of a directory so that it can be operated on by native services. Note in particular that readdir omits the extension and version when Unix report is in effect, so nothing should see the .DIR;1 that doesn't specifically ask for it by calling fileify.

All uses of fileify in the core are in [.vms]vms.c. In fact there are only three uses:

1.) Perl_flex_stat_int. This stores an expanded filename in native format in an extended version of the stat structure so that its callers have it available if needed, and if the file is a directory, it's fileified.

2.) do_rmdir. This is ultimately based on SYS$ERASE. The format in which filenames are reported to the user is irrelevant; SYS$ERASE still has to have the .DIR extension to be able to delete a directory.

3.) Perl_rename. This is based on LIB$RENAME_FILE. What I said about SYS$ERASE in #2 applies here too.

That's it. And in fact #2 seldom kicks in as rmdir normally gets the pre-fileified spec from stat() and only calls fileify directly if the stat failed.

It may be that we could do a vmsify first and then fileify, but I think I tried that once and ran into trouble. It may be that we could refactor some of this code so that fileify simply becomes a flag passed to int_tovmsspec or int_rmsexpand. There is certainly a great deal of redundant processing in these routines. But changing fileify to not do what it was designed to do would break (at least) rmdir and rename and probably other things that depend on a native fileified spec in the stat buffer.

> Neither DECC$EFS_CHARSET nor DECC$FILENAME_UNIX_REPORT has any impact
>> on this. So I've changed vmsify in Perl to do exactly the same thing
>> the CRTL does. Well, not exactly because the CRTL honors
>> DECC$FILENAME_UNIX_NO_VERSION by escaping the semicolon even if the
>> version spec is valid:
>
> Then Perl should also be using that feature.
>
>> $ DEFINE DECC$FILENAME_UNIX_NO_VERSION 1
>> $ mcr []to_vms
>> abc/xyz.dat;32767 Translating: abc/xyz.dat;32767 file:
>> [.ABC]XYZ.DAT^;32767 1 files found
>>
>> I haven't (yet) made Perl do this. It wouldn't be hard to do, but
>> I'm a little skeptical about trying to support the whole wilderness
>> of feature logicals.
>
> It is something that IMHO is critical to add, and I encountered this in the test harness for GNU make and had to modify the code to work around it.

OK, I'll add it.