Subject: Conformity and convenience problems with pathnames
Pascal J. Bourguignon
It is well known that implementations of CL pathnames have been greatly
implementation dependant. However, the standard still specifies clear
behavior for logical pathnames, for one thing, and for the other, since
there are several implementations working on the same POSIX systems
(unix including linux and MacOSX; and MS-Windows), it is desirable that
all implementations converge in their handling of pathnames on these
plateforms.
Personnaly, I resolved to use logical pathnames and logical-pathname
translations as much as possible, and to use make-pathname to build
portably physical pathnames.
However, most implementations have problems dealing with these two
aspects. To improve the situation, I wrote a little script to check
the behavior of implementations in these two aspects.
The script can be found at:
ftp://ftp.informatimago.com/users/pjb/lisp/check-pathnames.lisp
(tests specific to MS-Windows physical paths would need to be added).
I'm sending the results for each implementation on their own lists.
--
__Pascal Bourguignon__ http://www.informatimago.com/
A bad day in () is better than a good day in {}.
Juanjo
wrote:
> ftp://ftp.informatimago.com/users/pjb/lisp/check-pathnames.lisp
> (tests specific to MS-Windows physical paths would need to be added).
I have been working on this file and verifying whether ECL conforms to
its expectations. I find that the main problem with the file is the
interpretation of pathname case and more precisely, the meaning of
@':local' in this context.
So, for instance, in your script (pathname-name some-logical-
pathname :case :local) raises the expectation that the output should
be in the case of some local filesystem. That means :local has an
absolute meaning independent of the hostname. I would rather read
(pathname-* any-pathname :case :local) as getting the component (*) of
that pathname in the local case specified by its hostname.
If no hostname is available, then the default one. In other words, the
value of :case :local when retrieving should be the value associated
to that pathname's hostname (which may be the default one), and the
value of :case :local when building should be the value of that
pathname's hostname as specified by the arguments.
Given that you agree on this and change the script accordingly, with
the patches I will be pushing up to ECL's CVS repository soon, then
this implementation should conform to your expectations. See here:
;;; Loading "/Users/jjgarcia/build/ecl/check-pathnames.lisp"
check-pathnames of ECL (10.7.1)
================================================================================
Test and probe conforming logical pathnames, and their translation to
unix physical pathnames.
We want to check the good working of logical pathnames, and the
translation of logical pathnames to physical pathnames, in a
semi-standard way on unix systems.
Namely, given the logical host and its translations:
(setf (logical-pathname-translations "LOGICAL") nil)
(setf (logical-pathname-translations "LOGICAL")
'((#P"LOGICAL:**;*.*" #P"/tmp/**/*.*")
(#P"LOGICAL:**;*" #P"/tmp/**/*")))
#P"LOGICAL:DIR;SUBDIR;NAME.TYPE.NEWEST"
must be the same as (make-pathname :host "LOGICAL"
:directory '(:absolute "DIR"
"SUBDIR")
:name "NAME" :type
"TYPE" :version :newest
:case :common)
and must translate to: #P"/tmp/dir/subdir/name.type" on unix.
Merging physical pathnames specified with :case :common is also
tested:
(merge-pathnames (make-pathname :directory '(:relative "DIR"
"SUBDIR")
:name "NAME" :type
"TYPE" :version :newest
:case :common :default #1=#P"/tmp/")
#1# nil)
must give #P"/tmp/dir/subdir/name.type" on unix.
================================================================================
--------------------------------------------------------------------------------
Failed assertion: (IMPLY (EQL CUSTOMARY-CASE-1 :UPPER) (EQL CUSTOMARY-
CASE-2 :UPPER))
with: (EQL CUSTOMARY-CASE-1 :UPPER) = T
and: (EQL CUSTOMARY-CASE-2 :UPPER) = NIL
CUSTOMARY-CASE-1 = :UPPER
CUSTOMARY-CASE-2 = :LOWER
The customary case for the file system of ECL (10.7.1) seems to be
lower case.
*FEATURES* = (:UNIX :DARWIN :FORMATTER :LONG-LONG :UINT64-T :UINT32-
T :UINT16-T :RELATIVE-PACKAGE-NAMES :LONG-FLOAT :UNICODE :DFFI :CLOS-
STREAMS :CMU-FORMAT :ECL-PDE :DLOPEN :CLOS :BOEHM-GC :ANSI-CL :COMMON-
LISP :IEEE-FLOATING-POINT :PREFIXED-API :FFI :I686 :COMMON :ECL)
================================================================================
(MAKE-PATHNAME :HOST "LOGICAL" :DEVICE :UNSPECIFIC :DIRECTORY
(:ABSOLUTE "DIR" "SUBDIR") :NAME "NAME" :TYPE
"TYPE" :VERSION :NEWEST :CASE :COMMON)
LOGICAL-PATHNAME #P"LOGICAL:DIR;SUBDIR;NAME.TYPE.NEWEST"
-------------------- :case :local (default)
Host : "LOGICAL"
Device : :UNSPECIFIC
Directory : (:ABSOLUTE "DIR" "SUBDIR")
Name : "NAME"
Type : "TYPE"
Version : :NEWEST
-------------------- :case :common
Host : "LOGICAL"
Device : :UNSPECIFIC
Directory : (:ABSOLUTE "DIR" "SUBDIR")
Name : "NAME"
Type : "TYPE"
Version : :NEWEST
--------------------
CASE: COMMON
--------------------------------------------------------------------------------
Failed assertion: (STRING= (PATHNAME-HOST PATH :CASE :LOCAL) (POP
EXPECTED-VALUES))
with: (PATHNAME-HOST PATH :CASE :LOCAL) = "LOGICAL"
and: (POP EXPECTED-VALUES) = "logical"
19.2.2.1.2 makes no exception for pathname-host of logical pathnames.
--------------------------------------------------------------------------------
Failed assertion: (DIRLIST= (PATHNAME-DIRECTORY PATH :CASE :LOCAL)
(POP EXPECTED-VALUES))
with: (PATHNAME-DIRECTORY PATH :CASE :LOCAL) = (:ABSOLUTE "DIR"
"SUBDIR")
and: (POP EXPECTED-VALUES) = (:ABSOLUTE "dir" "subdir")
--------------------------------------------------------------------------------
Failed assertion: (STRING= (PATHNAME-NAME PATH :CASE :LOCAL) (POP
EXPECTED-VALUES))
with: (PATHNAME-NAME PATH :CASE :LOCAL) = "NAME"
and: (POP EXPECTED-VALUES) = "name"
--------------------------------------------------------------------------------
Failed assertion: (STRING= (PATHNAME-TYPE PATH :CASE :LOCAL) (POP
EXPECTED-VALUES))
with: (PATHNAME-TYPE PATH :CASE :LOCAL) = "TYPE"
and: (POP EXPECTED-VALUES) = "type"
================================================================================
(MAKE-PATHNAME :HOST "logical" :DEVICE :UNSPECIFIC :DIRECTORY
(:ABSOLUTE "dir" "subdir") :NAME "name" :TYPE
"type" :VERSION :NEWEST :CASE :LOCAL)
LOGICAL-PATHNAME #P"LOGICAL:DIR;SUBDIR;NAME.TYPE.NEWEST"
-------------------- :case :local (default)
Host : "LOGICAL"
Device : :UNSPECIFIC
Directory : (:ABSOLUTE "DIR" "SUBDIR")
Name : "NAME"
Type : "TYPE"
Version : :NEWEST
-------------------- :case :common
Host : "LOGICAL"
Device : :UNSPECIFIC
Directory : (:ABSOLUTE "DIR" "SUBDIR")
Name : "NAME"
Type : "TYPE"
Version : :NEWEST
--------------------
CASE: LOCAL
--------------------------------------------------------------------------------
Failed assertion: (STRING= (PATHNAME-HOST PATH :CASE :LOCAL) (POP
EXPECTED-VALUES))
with: (PATHNAME-HOST PATH :CASE :LOCAL) = "LOGICAL"
and: (POP EXPECTED-VALUES) = "logical"
19.2.2.1.2 makes no exception for pathname-host of logical pathnames.
--------------------------------------------------------------------------------
Failed assertion: (DIRLIST= (PATHNAME-DIRECTORY PATH :CASE :LOCAL)
(POP EXPECTED-VALUES))
with: (PATHNAME-DIRECTORY PATH :CASE :LOCAL) = (:ABSOLUTE "DIR"
"SUBDIR")
and: (POP EXPECTED-VALUES) = (:ABSOLUTE "dir" "subdir")
--------------------------------------------------------------------------------
Failed assertion: (STRING= (PATHNAME-NAME PATH :CASE :LOCAL) (POP
EXPECTED-VALUES))
with: (PATHNAME-NAME PATH :CASE :LOCAL) = "NAME"
and: (POP EXPECTED-VALUES) = "name"
--------------------------------------------------------------------------------
Failed assertion: (STRING= (PATHNAME-TYPE PATH :CASE :LOCAL) (POP
EXPECTED-VALUES))
with: (PATHNAME-TYPE PATH :CASE :LOCAL) = "TYPE"
and: (POP EXPECTED-VALUES) = "type"
#P"/Users/jjgarcia/build/ecl/check-pathnames.lisp"
Raymond Toy
> On Nov 21, 4:18 am, p...@informatimago.com (Pascal J. Bourguignon)
> wrote:
>> ftp://ftp.informatimago.com/users/pjb/lisp/check-pathnames.lisp
>> (tests specific to MS-Windows physical paths would need to be added).
>
> I have been working on this file and verifying whether ECL conforms to
> its expectations. I find that the main problem with the file is the
> interpretation of pathname case and more precisely, the meaning of
> @':local' in this context.
>
> So, for instance, in your script (pathname-name some-logical-
> pathname :case :local) raises the expectation that the output should
> be in the case of some local filesystem. That means :local has an
> absolute meaning independent of the hostname. I would rather read
> (pathname-* any-pathname :case :local) as getting the component (*) of
> that pathname in the local case specified by its hostname.
This seems reasonable. Should this be the expected behavior? AFAICT,
cmucl does this; the host determines the customary case for the
pathname. For physical pathnames, this is always lowercase. For
logical pathnames, the customary case is upcase (:common).
Ray
Juanjo
> On 11/26/10 8:18 PM, Juanjo wrote:
>
>
>
> > On Nov 21, 4:18 am, p...@informatimago.com (Pascal J. Bourguignon)
> > wrote:
> >>ftp://ftp.informatimago.com/users/pjb/lisp/check-pathnames.lisp
> >> (tests specific to MS-Windows physical paths would need to be added).
>
> > its expectations. I find that the main problem with the file is the
> > interpretation of pathname case and more precisely, the meaning of
> > @':local' in this context.
>
> > So, for instance, in your script (pathname-name some-logical-
> > pathname :case :local) raises the expectation that the output should
> > be in the case of some local filesystem. That means :local has an
> > absolute meaning independent of the hostname. I would rather read
> > (pathname-* any-pathname :case :local) as getting the component (*) of
> > that pathname in the local case specified by its hostname.
>
> This seems reasonable. Should this be the expected behavior? AFAICT,
> cmucl does this; the host determines the customary case for the
> pathname. For physical pathnames, this is always lowercase. For
> logical pathnames, the customary case is upcase (:common).
I don't know whether this is the "expected" behavior. The
specification is very ambiguous about the role of :case in accessors
vs. constructors. I could not find anything, but a couple of examples
in CLtL. However, in the end I just copied the idea of associating the
case of the host from CMUCL and SBCL, with the restriction that ECL
does not allow defining arbitrary hosts with non-Unix conventions --
this could be changed --.
In any case I can not yet produce a direct comparison between my
choices and SBCL's because they still seem to be different in other
aspects which make Pascal's script complain more often.
Raymond Toy
In case you didn't know cmucl fails the test because (pathname-host
<lpn> :case :local) returns an uppercase logical host name. Same is
true for pathname-directory, pathname-file and pathname-type.
Ray
Pascal J. Bourguignon
Now I think indeed, that (pathname-* p :case :local) should not be
applied on logical pathnames, to remain conforming a program. I'm
relaxing this from the my script, and will publish it soon with a more
complete answer. It seems to me now that the behavior the pathname
accessors on logical pathnames with :case :local (which unfortunately
is the default) is not specified, and therefore remains entirely
implementation dependant. On the other hand, I'm adding test cases for
TRANSLATE-PATHNAME, for which I'd expect a consistent behavior either to
or from logical pathnames and with unix physical pathnames.
> In any case I can not yet produce a direct comparison between my
> choices and SBCL's because they still seem to be different in other
> aspects which make Pascal's script complain more often.
Pascal J. Bourguignon
I'm relaxing this requirement, which on nth reading of CLHS, indeed is
not specified. I'll be happy to always using :case :common when
accessing logical pathnnames, to stay conforming.
Pascal J. Bourguignon
> On Nov 21, 4:18 am, p...@informatimago.com (Pascal J. Bourguignon)
> wrote:
>> ftp://ftp.informatimago.com/users/pjb/lisp/check-pathnames.lisp
>> (tests specific to MS-Windows physical paths would need to be added).
>
> I have been working on this file and verifying whether ECL conforms to
> its expectations. I find that the main problem with the file is the
> interpretation of pathname case and more precisely, the meaning of
> @':local' in this context.
>
> So, for instance, in your script (pathname-name some-logical-
> pathname :case :local) raises the expectation that the output should
> be in the case of some local filesystem. That means :local has an
> absolute meaning independent of the hostname. I would rather read
> (pathname-* any-pathname :case :local) as getting the component (*) of
> that pathname in the local case specified by its hostname.
>
> If no hostname is available, then the default one. In other words, the
> value of :case :local when retrieving should be the value associated
> to that pathname's hostname (which may be the default one), and the
> value of :case :local when building should be the value of that
> pathname's hostname as specified by the arguments.
Well, I think I don't want to impose that.
After re-reading chapter 19 again, it seems to me now that indeed, the
behavior of:
(pathname-name logical-pathname :case :local)
is unspecified, and while at first I had a reading of 19.2.2.1.2.1 which
let me to think the customary case was implementation-wide, it
definitely cannot be; eg. clhs translate-pathname says:
translate-pathname maps customary case in source into customary case
in the output pathname.
Therefore 'host' in 19.2.2.1.2.1 must indeed be interpreted as the
pathname-host of the pathname:
For the functions in Figure 19-2, a value of :local for the :case
argument (the default for these functions) indicates that the
functions should receive and yield strings in component values as if
they were already represented according to the host file system's
convention for case.
Therefore, I shall always use :case :common on logical pathnames.
The notion of 'customary case' doesn't apply to logical hosts. It is a
property of the file systems of the physical hosts.
Logical pathnames from a single logical host can be translated to
different physical (or logical) pathnames or even none at all.
Imposing a specific behavior on :case :local for logical pathnames would
require the implementations either:
- to define a default customary case (eg. lowercase for unix), or
- to define a customary case for logical pathnames (eg. uppercase), or
- to do a logical pathname translation (possibly a chain of translations
even), to possibly eventually find a physical pathname, to find the
customary case of the underlying file system. This would be
problematic since first there could be no physical pathname at all,
eg. (setf (logical-pathname-translations "LOGHOST") '()) or if there
is a pathname, there could be no file system mounted (yet) there.
[It is probably an error to define logical pathname translations that
loop between logical pathnames without resolving to a physical
pathname, since an iterative process is defined for
translate-logical-pathname with that stop condition].
Conforming programs can avoid the question by using always :case :common
when accessing logical pathname components.
> Given that you agree on this and change the script accordingly,
I updated the script, relaxing the requirements on pathname accessors,
which are not called for logical-pathnames with :case :local anymore.
> with the patches I will be pushing up to ECL's CVS repository soon,
> then this implementation should conform to your expectations. See
> here:
> [...same results as those I sent for ecl...]
The new script is at:
ftp://ftp.informatimago.com/users/pjb/lisp/check-pathnames/check-pathnames.lisp
I also upgraded the script to format the output as reStructured Text, in
an intend to improve it's readability (at least, when converted to PDF,
I hope).
I added checks of TRANSLATE-PATHNAME, since this is used by
TRANSLATE-LOGICAL-PATHNAME. Almost all the implementations also have
problems with that, even in simple cases...
There's also a check-all shell script to run it on all the CL
implementations installed, and the results I have with my current
implementations are published in:
ftp://ftp.informatimago.com/users/pjb/lisp/check-pathnames/
I also put it in a git repository:
git clone http://git.informatimago.com/public/misc/check-pathnames