PEP on path module for standard library
Michael Hoffman
<http://www.jorendorff.com/articles/python/path/>, which is frequently
recommended here on c.l.p. I submitted an RFE to add it to the Python
standard library, and Reinhold Birkenfeld started a discussion on it in
python-dev
<http://mail.python.org/pipermail/python-dev/2005-June/054438.html>.
The upshot of the discussion was that many python-dev'ers wanted path
added to the stdlib, but Guido was not convinced and said it must have a
PEP. So Reinhold and I are going to work on one. Reinhold has already
made some changes to the module to fit the python-dev discussion and put
it in CPython CVS at nondist/sandbox/path.
For the PEP, do any of you have arguments for or against including path?
Code samples that are much easier or more difficult with this class
would also be most helpful.
I use path in more of my modules and scripts than any other third-party
module, and I know it will be very helpful when I no longer have to
worry about deploying it.
Thanks in advance,
--
Michael Hoffman
Peter Hansen
> For the PEP, do any of you have arguments for or against including path?
> Code samples that are much easier or more difficult with this class
> would also be most helpful.
I believe the strongest argument for "path" can be made for how it
integrates functionality which, although closely related conceptually,
is currently distributed across a half dozen or more different modules
in the standard library. Especially for newbies (I can only imagine, at
this stage) it would make working with files much easier in a many ways.
"Easier" or "more difficult" is a subjective thing, of course, but one
can't argue with the fact that path can sometimes do through a single
object what would otherwise require several imports and a bunch of calls
into things like open(), os.path, grep, and shutil.
Examples showing effective uses of path that simplify those cases would
probably merit the label "easier" even in Guido's mind, though
unfortunately that's not certain. "Easier" in some minds might simply
translate to "many lines less code", and while path can sometimes do
that, aside from the ease of splitting and joining stuff without
multiple calls to os.path.this-and-that, it really doesn't often reduce
code size _that_ much, in my experience. (Postings to c.l.p showing a
50% reduction in code size for contrived examples notwithstanding.)
A related thoughts: since paths are objects, they have attributes or
properties, and having things like ".basename" and ".parent" readily
available without having to do obscure things like
os.path.split(somepath)[0] makes things much easier to read (therefore
more maintainable). In fact, I'd propose that as another strong
argument in path's favour: it makes code much more readable, even if not
"easier" to write.
Hmm... does "easier" or "more difficult" apply to the writing of the
code or the reading of it? I find it self-evident that code written
using "path" is much easier to read, not necessarily much easier to
write (for non-newbies).
I'd summarize this by saying that the integration of "path" in the
stdlib would make it easier for newbies to write code (that might not be
obvious to a non-newbie... shall we ask some to help?), and easier for
everyone to read code (self-evident, no?), and if that's not a
sufficient condition for inclusion I don't know what is.
-Peter
Fernando Perez
> Michael Hoffman wrote:
>> For the PEP, do any of you have arguments for or against including path?
>> Code samples that are much easier or more difficult with this class
>> would also be most helpful.
>
> I believe the strongest argument for "path" can be made for how it
> integrates functionality which, although closely related conceptually,
> is currently distributed across a half dozen or more different modules
> in the standard library. Especially for newbies (I can only imagine, at
> this stage) it would make working with files much easier in a many ways.
+10
One of the few things that annoys me about the stdlib is what one could call
performing 'shell-scripting-like' tasks, and precisely because of the problem
you point out. A number of conceptually related and common tasks are
scattered all over, and every time I need to write this kind of code, I find
myself paging over the docs for multiple modules, with no real intuition as to
where I could even guess where to find things. This is very unusual for
python, where in most cases things are so well organized, that blind guessing
tends to work remarkably well.
Personally I like the path module _a lot_, though I'm sure a thorough once-over
from c.l.py and python-dev, via a PEP, can only make it better and smooth out
hidden rough edges and corner cases. But I'll be very happy if it does go
into the stdlib in the future.
Just my .02.
Best,
f
mk...@webmd.net
problem with using it. When you try to 'freeze' an application module,
and Jason's 'path' module is present in any of the directories that are
looked at by freeze's module finder (your app doesn't have to import
it), freeze goes into an infinite loop of imports, eventually getting a
'maximum recursion depth' exception. This seems to be related to
freeze getting confused between 'os.path' and Jason's 'path'.
I encountered this using Jason's latest 'path' module and Python 2.3.2.
I was able to solve it for my use by renaming path.py to newpath.py
and using 'from newpath import path' in my modules.
I've just notified Jason about this. I presume a solution like mine
will be used, and look forward to seeing Jason's module in stdlib.
Reinhold Birkenfeld
from python-dev, and put the result in the Python CVS tree under
nondist/sandbox/path.
Most prominent change is that it doesn't inherit from str/unicode anymore.
I found this distinction important, because as a str subclass the Path object
has many methods that don't make sense for it.
Peter Hansen wrote:
> Michael Hoffman wrote:
>> For the PEP, do any of you have arguments for or against including path?
>> Code samples that are much easier or more difficult with this class
>> would also be most helpful.
>
> I believe the strongest argument for "path" can be made for how it
> integrates functionality which, although closely related conceptually,
> is currently distributed across a half dozen or more different modules
> in the standard library. Especially for newbies (I can only imagine, at
> this stage) it would make working with files much easier in a many ways.
>
> "Easier" or "more difficult" is a subjective thing, of course, but one
> can't argue with the fact that path can sometimes do through a single
> object what would otherwise require several imports and a bunch of calls
> into things like open(), os.path, grep, and shutil.
Correct.
> Examples showing effective uses of path that simplify those cases would
> probably merit the label "easier" even in Guido's mind, though
> unfortunately that's not certain. "Easier" in some minds might simply
> translate to "many lines less code", and while path can sometimes do
> that, aside from the ease of splitting and joining stuff without
> multiple calls to os.path.this-and-that, it really doesn't often reduce
> code size _that_ much, in my experience. (Postings to c.l.p showing a
> 50% reduction in code size for contrived examples notwithstanding.)
Well, these examples are the ones we'd like to see here. So, people:
If you posted examples to c.l.py in the past, please try to collect
them here!
> A related thoughts: since paths are objects, they have attributes or
> properties, and having things like ".basename" and ".parent" readily
> available without having to do obscure things like
> os.path.split(somepath)[0] makes things much easier to read (therefore
> more maintainable). In fact, I'd propose that as another strong
> argument in path's favour: it makes code much more readable, even if not
> "easier" to write.
>
> Hmm... does "easier" or "more difficult" apply to the writing of the
> code or the reading of it? I find it self-evident that code written
> using "path" is much easier to read, not necessarily much easier to
> write (for non-newbies).
And it is much more "Pythonic" in my eyes. Though that word may be inaccurate
when it comes from someone else that Guido, I feel that endless chains of
'[os.path.join(os.path.join(z, "a"), x) for x in os.path.listdir(os.path.join(z, "a") if os.path.isfile(os.path.join(....
are not qualified as being Pythonic.
> I'd summarize this by saying that the integration of "path" in the
> stdlib would make it easier for newbies to write code (that might not be
> obvious to a non-newbie... shall we ask some to help?), and easier for
> everyone to read code (self-evident, no?), and if that's not a
> sufficient condition for inclusion I don't know what is.
Reinhold
Terry Reedy
"Reinhold Birkenfeld" <reinhold-birk...@wolke7.net> wrote in
message news:3kacleF...@individual.net...
> Most prominent change is that it doesn't inherit from str/unicode
> anymore.
> I found this distinction important, because as a str subclass the Path
> object
> has many methods that don't make sense for it.
While I am - on adding more to learn, I am + on collecting scattered
filesystem functions into methods of coherent classes for eventually
replacement of the former. And I understand that this will mean a period
of duplication. But it is important to get the replacement right.
My only knowledge of the path module is what has been posted. However, it
seems to me that a path is conceptually a sequence of strings, rather than
the single joined-string representation thereof. If so, then of course it
is not and should not be a subclass of single strings. But that Path was
so defined makes me wonder, in my ignorance, whether the current
implementation is the best we can do for the future.
One advantage, for instance, of a split-up list implementation is that a
set of paths with a common prefix could be represented by replacing the
last string with a set of strings.
A more OO-friendly OS than the dominant ones today would accept a path as a
list (sequence) instead of requiring that the list be joined (by an
artifactual character) just to be split back into a list again.
My thoughts anyway.
Terry J. Reedy
Michael Hoffman
This sounds like a bug in "freeze" rather than something that should be
worked around in the standard library. Although there have already been
people opposed to naming it path because the duplication with os.path
might confuse humans.
--
Michael Hoffman
Michael Hoffman
> FYI: I modified the path module a bit so that it fits many of the suggestions
> from python-dev, and put the result in the Python CVS tree under
> nondist/sandbox/path.
>
> Most prominent change is that it doesn't inherit from str/unicode anymore.
> I found this distinction important, because as a str subclass the Path object
> has many methods that don't make sense for it.
Having path descend from str/unicode is extremely useful since I can
then pass a path object to any function someone else wrote without
having to worry about whether they were checking for basestring. I think
there is a widely used pattern of accepting either a basestring[1] or a
file-like object as a function argument, and using isinstance() to
figure out which it is.
What do you gain from removing these methods? A smaller dir()?
[1] Probably str in actuality.
--
Michael Hoffman
Reinhold Birkenfeld
> Reinhold Birkenfeld wrote:
>> FYI: I modified the path module a bit so that it fits many of the suggestions
>> from python-dev, and put the result in the Python CVS tree under
>> nondist/sandbox/path.
>>
>> Most prominent change is that it doesn't inherit from str/unicode anymore.
>> I found this distinction important, because as a str subclass the Path object
>> has many methods that don't make sense for it.
>
> Having path descend from str/unicode is extremely useful since I can
> then pass a path object to any function someone else wrote without
> having to worry about whether they were checking for basestring. I think
> there is a widely used pattern of accepting either a basestring[1] or a
> file-like object as a function argument, and using isinstance() to
> figure out which it is.
Where do you see that pattern? IIRC it's not in the stdlib.
> What do you gain from removing these methods? A smaller dir()?
It made sense to me at the time I changed this, although at the moment
I can't exactly recall the reasons.
Probably as Terry said: a path is both a list and a string.
Reinhold
Reinhold Birkenfeld
As the most likely placement will be a class named "Path" inside the "os.path"
module, that bug with freeze won't apply to the "stdlib version" of Path.
Reinhold
John Roth
news:dbofk2$g1f$1...@gemini.csx.cam.ac.uk...
> Many of you are familiar with Jason Orendorff's path module
> <http://www.jorendorff.com/articles/python/path/>, which is frequently
> recommended here on c.l.p. I submitted an RFE to add it to the Python
> standard library, and Reinhold Birkenfeld started a discussion on it in
> python-dev
> <http://mail.python.org/pipermail/python-dev/2005-June/054438.html>.
>
> The upshot of the discussion was that many python-dev'ers wanted path
> added to the stdlib, but Guido was not convinced and said it must have a
> PEP.
Why did Guido want a PEP? Is it because he likes the idea but
feels the feature set needs to be examined a bit more by the wider
community, or is it some other reason?
I'm all in favor of having something that gives an alternative to
the kludge of functions that are "just a thin wrapper on the C
standard library." Considering the known problems with the
C standard library and the fact that it's strictly procedural, that
statement doesn't fill me with confidence. Rather it creates a
mild sense of dread: nobody has thought out how to do those
functions in a useful oo manner.
Path looks useable to me. Do I think it's going to be the
last word? I sincerely hope not! The only way we're going
to find out where it really needs to go from here, though, is
to put it out and find out how the wider community uses
and abuses it.
John Roth
Reinhold Birkenfeld
> "Michael Hoffman" <cam....@mh391.invalid> wrote in message
> news:dbofk2$g1f$1...@gemini.csx.cam.ac.uk...
>> Many of you are familiar with Jason Orendorff's path module
>> <http://www.jorendorff.com/articles/python/path/>, which is frequently
>> recommended here on c.l.p. I submitted an RFE to add it to the Python
>> standard library, and Reinhold Birkenfeld started a discussion on it in
>> python-dev
>> <http://mail.python.org/pipermail/python-dev/2005-June/054438.html>.
>>
>> The upshot of the discussion was that many python-dev'ers wanted path
>> added to the stdlib, but Guido was not convinced and said it must have a
>> PEP.
>
> Why did Guido want a PEP? Is it because he likes the idea but
> feels the feature set needs to be examined a bit more by the wider
> community, or is it some other reason?
He said,
"""
Whoa! Do we really need a completely different mechanism for doing the
same stuff we can already do? The path module seems mostly useful for
folks coming from Java who are used to the Java Path class. With the
massive duplication of functionality we should also consider what to
recommend for the future: will the old os.path module be deprecated,
or are we going to maintain both alternatives forever? (And what about
all the duplication with the os module itself, like the cwd()
constructor?) Remember TOOWTDI.
"""
Reinhold
Michael Hoffman
> Michael Hoffman wrote:
>
>>Having path descend from str/unicode is extremely useful since I can
>>then pass a path object to any function someone else wrote without
>>having to worry about whether they were checking for basestring. I think
>>there is a widely used pattern of accepting either a basestring[1] or a
>>file-like object as a function argument, and using isinstance() to
>>figure out which it is.
>
> Where do you see that pattern? IIRC it's not in the stdlib.
I do not think it is a *good* pattern, but it is used in Biopython. Of
course, there they ARE using things like type("") so on a unicode
filesystem it would already break. I seem to recall seeing it elsewhere,
but I can't remember where.
If you remove the basestring superclass, then you remove the ability to
use path objects as a drop-in replacement for any path string right now.
You will either have to use str(pathobj) or carefully check that the
function/framework you are passing the path to does not use isinstance()
or any of the string methods that are now gone.
>>What do you gain from removing these methods? A smaller dir()?
>
> It made sense to me at the time I changed this, although at the moment
> I can't exactly recall the reasons.
>
> Probably as Terry said: a path is both a list and a string.
I can see the case for thinking of it in both of those ways. In the end
a path is a sequence object. But a sequence of what?
I have a path that looks like this:
r"c:\windows\system32:altstream\test.dir\myfile.txt.zip:altstream"
One way to divide this is solely based on path separators:
['c:', 'windows', 'system32:altstream', 'test.dir',
'myfile.txt.zip:altstream']
But then some of the elements of this sequence have more meaning than
just being strings. "c:" is certainly something different from
"windows." The file name and alternate data stream name of each element
could be represented as a tuple.
The extensions can also be dealt with as a sequence. I have dealt with
things like filename = "filename.x.y.z" and wanted to get "filename.x"
before. The current stdlib solution,
os.path.splitext(os.path.splitext(filename)[0])[0] is extremely clunky,
and I have long desired something better. (OK, using
filename.split(os.extsep) works a little better, but you get the idea.)
So if you start breaking the path into a sequence of bigger items than
single character, where does it stop? What is a good design for this?
--
Michael Hoffman
Michael Hoffman
> John Roth wrote:
>
>>Why did Guido want a PEP?
>
>
> """
> Whoa! Do we really need a completely different mechanism for doing the
> same stuff we can already do? The path module seems mostly useful for
> folks coming from Java who are used to the Java Path class.
> """
What is this Java Path class? I have been STFWing and have found nothing
on it in the. Indeed if you search for "Java Path class" (with quotes)
almost half of the pages are this message from Guido. ;)
Any Java hackers here want to tell us of the wonders of the Java Path
class? I would be interested in seeing how other OO languages deal with
paths.
--
Michael Hoffman
John Roth
Read literally, this says (at least to me) "I don't want to fix it because
I don't think it's broke."
As far as the Java remark is concerned, I suspect that it's because
in Java there is no such thing as a function; everything is either a
method on an object or a static method on a class.
And as far as I'm concerned, it's broke. I could see getting rid of the
bits and pieces of procedural code in 3.0, but not sooner.
John Roth
>
> Reinhold
George Sakkis
> > Why did Guido want a PEP? Is it because he likes the idea but
> > feels the feature set needs to be examined a bit more by the wider
> > community, or is it some other reason?
>
> He said,
>
> """
> Whoa! Do we really need a completely different mechanism for doing the
> same stuff we can already do? The path module seems mostly useful for
> folks coming from Java who are used to the Java Path class. With the
> massive duplication of functionality we should also consider what to
> recommend for the future: will the old os.path module be deprecated,
> or are we going to maintain both alternatives forever? (And what about
> all the duplication with the os module itself, like the cwd()
> constructor?) Remember TOOWTDI.
> """
Duplication is a valid point for debate, so the PEP should definitely address it. IMO os.path and
most (if not all) other equivalent modules and functions should be deprecated, though still working
until 2.9 for backwards compatibility, and dropped for python 3K.
George
George Sakkis
> "Reinhold Birkenfeld" <reinhold-birk...@wolke7.net> wrote
> > He said,
> >
> > """
> > Whoa! Do we really need a completely different mechanism for doing the
> > same stuff we can already do? The path module seems mostly useful for
> > folks coming from Java who are used to the Java Path class. With the
> > massive duplication of functionality we should also consider what to
> > recommend for the future: will the old os.path module be deprecated,
> > or are we going to maintain both alternatives forever? (And what about
> > all the duplication with the os module itself, like the cwd()
> > constructor?) Remember TOOWTDI.
> > """
>
> Read literally, this says (at least to me) "I don't want to fix it because
> I don't think it's broke."
Or rather "I prefer a single existing mediocre solution than two solutions (even if the second was
better)".
George
Neil Hodgson
> And it is much more "Pythonic" in my eyes. Though that word may be inaccurate
> when it comes from someone else that Guido, I feel that endless chains of
> '[os.path.join(os.path.join(z, "a"), x) for x in os.path.listdir(os.path.join(z, "a") if os.path.isfile(os.path.join(....
> are not qualified as being Pythonic.
I like Path but the above is trying too hard to be poor code.
os.path.join takes more than 2 arguments, so that should be
[os.path.join(z, "a", x) for x in os.path.listdir(os.path.join(z, "a")
if os.path.isfile(os.path.join(....
Neil
Andrew Dalke
> Having path descend from str/unicode is extremely useful since I can
> then pass a path object to any function someone else wrote without
> having to worry about whether they were checking for basestring. I think
> there is a widely used pattern of accepting either a basestring[1] or a
> file-like object as a function argument, and using isinstance() to
> figure out which it is.
Reinhold Birkenfeld wrote:
> Where do you see that pattern? IIRC it's not in the stdlib.
Here's the first place that comes to mind for me
xml.sax.saxutils
def prepare_input_source(source, base = ""):
"""This function takes an InputSource and an optional base URL and
returns a fully resolved InputSource object ready for reading."""
if type(source) in _StringTypes:
source = xmlreader.InputSource(source)
elif hasattr(source, "read"):
f = source
source = xmlreader.InputSource()
source.setByteStream(f)
if hasattr(f, "name"):
source.setSystemId(f.name)
and xml.dom.pulldom
def parse(stream_or_string, parser=None, bufsize=None):
if bufsize is None:
bufsize = default_bufsize
if type(stream_or_string) in _StringTypes:
stream = open(stream_or_string)
else:
stream = stream_or_string
if not parser:
parser = xml.sax.make_parser()
return DOMEventStream(stream, parser, bufsize)
Using the power of grep
aifc.py
def __init__(self, f):
if type(f) == type(''):
f = __builtin__.open(f, 'rb')
# else, assume it is an open file object already
self.initfp(f)
binhex.py
class HexBin:
def __init__(self, ifp):
if type(ifp) == type(''):
ifp = open(ifp)
imghdr.py
if type(file) == type(''):
f = open(file, 'rb')
h = f.read(32)
else:
location = file.tell()
h = file.read(32)
file.seek(location)
f = None
mimify.py
if type(infile) == type(''):
ifile = open(infile)
if type(outfile) == type('') and infile == outfile:
import os
d, f = os.path.split(infile)
os.rename(infile, os.path.join(d, ',' + f))
else:
ifile = infile
wave.py
def __init__(self, f):
self._i_opened_the_file = None
if type(f) == type(''):
f = __builtin__.open(f, 'rb')
self._i_opened_the_file = f
# else, assume it is an open file object already
self.initfp(f)
compiler/transformer.py:
if type(file) == type(''):
file = open(file)
return self.parsesuite(file.read())
plat-mac/applesingle.py
if type(input) == type(''):
input = open(input, 'rb')
# Should we also test for FSSpecs or FSRefs?
header = input.read(AS_HEADER_LENGTH)
site-packages/ZODB/ExportImport.py
if file is None: file=TemporaryFile()
elif type(file) is StringType: file=open(file,'w+b')
site-packages/numarray/ndarray.py
if type(file) == type(""):
name = 1
file = open(file, 'wb')
site-packages/kiva/imaging/GdImageFile.py
if type(fp) == type(""):
import __builtin__
filename = fp
fp = __builtin__.open(fp, "rb")
else:
filename = ""
site-packages/reportlab/graphics/renderPM.py
if type(image.path) is type(''):
im = _getImage().open(image.path).convert('RGB')
else:
im = image.path.convert('RGB')
site-packages/twisted/protocols/irc.py
def __init__(self, file):
if type(file) is types.StringType:
self.file = open(file, 'r')
(hmm, that last one looks buggy. It should
have a "else: self.file = file" afterwards.)
Used in the std. lib and used by many different
people. (I excluded the Biopython libraries
in this list, btw, because I may have influenced
the use of this sort of type check.)
Andrew
da...@dalkescientific.com
Stefan Rank
> Reinhold Birkenfeld wrote:
>
>>John Roth wrote:
>>
>>
>>>Why did Guido want a PEP?
>>
>>He said,
>>
>>"""
>>Whoa! Do we really need a completely different mechanism for doing the
>>same stuff we can already do? The path module seems mostly useful for
>>folks coming from Java who are used to the Java Path class.
> > """
>
> What is this Java Path class? I have been STFWing and have found nothing
> on it in the. Indeed if you search for "Java Path class" (with quotes)
> almost half of the pages are this message from Guido. ;)
>
> Any Java hackers here want to tell us of the wonders of the Java Path
> class?
no such thing exists.
there is only the `File` class that incorporates a little bit of the
`path` functionality and some of the python built-in `file` functionality.
my little self would actually propose to make *path* a built-in type as
an intermediate between file and basestring/str/unicode (the latter is
probably needed).
*ducks*
Michael Hoffman
> on 22.07.2005 00:21 Michael Hoffman said the following:
>> class?
>
> no such thing exists.
>
> there is only the `File` class that incorporates a little bit of the
> `path` functionality and some of the python built-in `file` functionality.
Ah, here it is:
http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html
--
Michael Hoffman
Duncan Booth
>> Read literally, this says (at least to me) "I don't want to fix it
>> because I don't think it's broke."
>
> Or rather "I prefer a single existing mediocre solution than two
> solutions (even if the second was better)".
>
Except that he is open to persuasion, so the PEP has to demonstrate that
the duplication is worth the benefit.
Personally I think the concept of a specific path type is a good one, but
subclassing string just cries out to me as the wrong thing to do. In other
words, to me a path represents something in a filesystem, the fact that it
has one, or indeed several string representations does not mean that the
path itself is simply a more specific type of string.
You should need an explicit call to convert a path to a string and that
forces you when passing the path to something that requires a string to
think whether you wanted the string relative, absolute, UNC, uri etc.
It may even be that we need a hierarchy of path classes: URLs need similar
but not identical manipulations to file paths, so if we want to address the
failings of os.path perhaps we should also look at the failings of urlparse
at the same time.
Michael Hoffman
> You should need an explicit call to convert a path to a string and that
> forces you when passing the path to something that requires a string to
> think whether you wanted the string relative, absolute, UNC, uri etc.
Egad. I'm not sure if that will really make people's lives easier.
--
Michael Hoffman
Thomas Heller
That was a bug in modulefinder, which was fixed in Python 2.4 and 2.3.4.
See http://www.python.org/sf/876278.
Thomas
Harald Armin Massa
>and Jason's 'path' module is present in any of the directories that are
>looked at by freeze's module finder (your app doesn't have to import
>it), freeze goes into an infinite loop of imports, eventually getting a
>'maximum recursion depth' exception. This seems to be related to
>freeze getting confused between 'os.path' and Jason's 'path'.
This is a bug in distutils. Thomas Hellers py2exe encounters the same
bug. As much as I remember our conversation, he submitted a patch to
distutils.
In the meanwhile I renamed path.py to jpath.py, usings Jason's first
letter in a motion of honour while circumventing this bug.
Harald
Harald Armin Massa
>then pass a path object to any function someone else wrote without
>having to worry about whether they were checking for basestring.
specialities of path dealing.
Being able to use path-opjects at every place where I would use str or
unicode is very essential, because I often use Python to tame Excel and
Word. To open files within these programms needs some "plain str" as
"PATH" for the file. (which, of course, can also be down by ways to
"convert" PATH to STRING.
Harald
Peter Hansen
> Michael Hoffman wrote:
>
> It made sense to me at the time I changed this, although at the moment
> I can't exactly recall the reasons.
Along with some of the others (and as a fairly heavy user of "path"), I
would caution strongly against jumping to do make this change.
Given that a strong part of the justification for path's inclusion in
the standard library (as expressed here in the past) is that it is
stable and widely used, making such a fundamental change at this late
stage just prior to its possible acceptance seems to me very risky.
I have noticed in a number of cases where a "path" was usable as a
drop-in replacement for strings that previously contained paths. I
can't say for sure, but I strongly suspect some of that could would be
broken if "paths" weren't basestrings. I'll attempt to check in my own
code.
Even if those uses don't actually break, it can *also* be useful to have
the string methods available on a path object, so I'm also uncertain
what you gain by removing that connection, though it's clear what things
you might be losing.
-2 for this idea.
-Peter
Peter Hansen
> Personally I think the concept of a specific path type is a good one, but
> subclassing string just cries out to me as the wrong thing to do. In other
> words, to me a path represents something in a filesystem, the fact that it
> has one, or indeed several string representations does not mean that the
> path itself is simply a more specific type of string.
>
> You should need an explicit call to convert a path to a string and that
> forces you when passing the path to something that requires a string to
> think whether you wanted the string relative, absolute, UNC, uri etc.
Duncan, are you another formerly non-user of path who has this opinion,
or have you already attempted to use path extensively in your code?
I'm not saying I dismiss the opinions of those who haven't actually
tried working with a string-based path object, but it's worth
considering that you might adopt a different opinion after using it for
a while.
I did.
-Peter
George Sakkis
> Personally I think the concept of a specific path type is a good one, but
> subclassing string just cries out to me as the wrong thing to do. In other
> words, to me a path represents something in a filesystem, the fact that it
> has one, or indeed several string representations does not mean that the
> path itself is simply a more specific type of string.
>
> You should need an explicit call to convert a path to a string and that
> forces you when passing the path to something that requires a string to
> think whether you wanted the string relative, absolute, UNC, uri etc.
First off, I find this is a relatively small detail overall, that is,
regardless of whether path subclasses string or not, its addition in
the standard library will be a step forward. Havind said that, I think
the choice between subclassing or not is going to be a
practicality-vs-purity decision. You're right, conceptually a path
HAS_A string description, not IS_A string, so from a pure OO point of
view, it should not inherit string. OTOH, people in favor of the
subclassing point out the convenience for many (or most) common cases.
It's a tradeoff, so arguments for both cases should be discussed.
George
John Roth
"Duncan Booth" <duncan...@invalid.invalid> wrote in message
news:Xns969B63FB97...@127.0.0.1...
> George Sakkis wrote:
>
>
> You should need an explicit call to convert a path to a string and that
> forces you when passing the path to something that requires a string to
> think whether you wanted the string relative, absolute, UNC, uri etc.
>
> It may even be that we need a hierarchy of path classes: URLs need similar
> but not identical manipulations to file paths, so if we want to address
> the
> failings of os.path perhaps we should also look at the failings of
> urlparse
> at the same time.
You have to start somewhere. One of the lessons that's beginning
to seep into people's minds is that getting something that works
out there is almost always preferable to (over) design by committee.
How to do a comprehensive, covers all the corner cases file
system object (or object hierarchy, etc) has been discussed before,
and nothing has ever come of it. Starting with an object that
actually does something some people want gives the designers a
chance to look at things in the wild.
John Roth
Stefan Rank
> "Duncan Booth" <duncan...@invalid.invalid> wrote:
>
>>Personally I think the concept of a specific path type is a good one, but
>>subclassing string just cries out to me as the wrong thing to do. In other
>>words, to me a path represents something in a filesystem, the fact that it
>>has one, or indeed several string representations does not mean that the
>>path itself is simply a more specific type of string.
>>
> practicality-vs-purity decision. You're right, conceptually a path
> HAS_A string description, not IS_A string, so from a pure OO point of
> view, it should not inherit string.
Java has `File` which mixes the concepts "an object in the filesystem"
and "a structured locator for such objects (in a hierarchical fs) that
might or might not correspond to an object that is actually there".
`file` and `path` separate that. I think this is very reasonable.
(It would be nice to get `path`(s) easily from a `file`, at the moment
there is only file.name if I'm not mistaken).
And a `path`, to me, actually IS_A string (unicode string) that happens
to have special behaviour (such as os dependent quirks like a
pathseparator that automatically get translated, comparable to '\n' used
internally in strings translated to '\n'|'\r\n')
stefan
Michael Hoffman
> Havind said that, I think
> the choice between subclassing or not is going to be a
> practicality-vs-purity decision. You're right, conceptually a path
> HAS_A string description, not IS_A string, so from a pure OO point of
> view, it should not inherit string. OTOH, people in favor of the
> subclassing point out the convenience for many (or most) common cases.
It would be an entirely different matter if we were designing a language
from scratch. But we have to deal with an existing codebase that expects
strings.
Here's some code I just wrote seconds ago to construct a path for a scp
upload:
"""
DST_DIRPATH = path("host:~/destination")
RSS_EXT = "rss"
dst_filenamebase = os.extsep.join([postcode.lower(), RSS_EXT])
dst_filepath = DST_DIRPATH.joinpath(dst_filenamebase)
"""
With the current path implementation, this Just Works. If I were using
something that parsed and understood paths, the scp/rcp convention of
host:filename would either cause an error or have to be programmed in
separately. The current implementation is much more flexible.
What are the practical advantages and conveniences of *not* subclassing
from basestring?
--
Michael Hoffman
George Sakkis
> Personally I think the concept of a specific path type is a good one, but
> subclassing string just cries out to me as the wrong thing to do. In other
> words, to me a path represents something in a filesystem, the fact that it
> has one, or indeed several string representations does not mean that the
> path itself is simply a more specific type of string.
>
> You should need an explicit call to convert a path to a string and that
> forces you when passing the path to something that requires a string to
> think whether you wanted the string relative, absolute, UNC, uri etc.
First off, I find this is a relatively small detail overall, that is, regardless of whether path
subclasses string or not, its addition in the standard library will be a step forward. Havind said
that, I think the choice between subclassing or not is going to be a practicality-vs-purity
decision. You're right, conceptually a path HAS_A string description, not IS_A string, so from a
pure OO point of view, it should not inherit string. OTOH, people in favor of the subclassing point
out the convenience for many (or most) common cases. It's a tradeoff, so arguments for both cases
should be discussed.
George
Duncan Booth
> Duncan, are you another formerly non-user of path who has this opinion,
> or have you already attempted to use path extensively in your code?
I'm a currently non-user of path who would probably use it if it were in
the standard library but so far have been satisfied to use os.path.
> I'm not saying I dismiss the opinions of those who haven't actually
> tried working with a string-based path object, but it's worth
> considering that you might adopt a different opinion after using it for
> a while.
I fully accept that. My point is simply that as a non-user, it sounds to me
as though subclassing string is the wrong approach. I would have expected a
path object to be a sequence of path elements rather than a sequence of
characters. This is basically just a gut feeling though, so I'm perfectly
happy to be told that I'm wrong.
BTW, does it matter at all in practical use that the base class of path
varies between str and unicode depending on the platform?
John Roth wrote:
> You have to start somewhere. One of the lessons that's beginning
> to seep into people's minds is that getting something that works
> out there is almost always preferable to (over) design by committee.
Dead right, but once it goes into the standard library it has to pretty
well stop evolving, so it needs to be right, or as close as possible before
that happens.
Daniel Dittmar
> I would have expected a
> path object to be a sequence of path elements rather than a sequence of
> characters.
Maybe it's nitpicking, but I don't think that a path object should be a
'sequence of path elements' in an iterator context.
This means that
for element in pathobject:
has no intuitive meaning for me, so it shouldn't be allowed.
Daniel
John Roth
>
> John Roth wrote:
>> You have to start somewhere. One of the lessons that's beginning
>> to seep into people's minds is that getting something that works
>> out there is almost always preferable to (over) design by committee.
>
> Dead right, but once it goes into the standard library it has to pretty
> well stop evolving, so it needs to be right, or as close as possible
> before
> that happens.
It has to stop evolving in incompatible directions, at least. Although
there is a precident with the process functions, classes, module,
whatever it is. It's up to five versions now, isn't it?
AFAICT, from a very broad brush perspective, there is really
only one substantive issue: how to handle multiple path-like
"things". URLs have been mentioned in this thread, different
file systems and a possible in-memory file system have been
mentioned in other threads.
So whatever gets out there first shouldn't preempt the ability
to eventually fit into a wider structure without substantial
and incompatible modifications.
John Roth
>
John Roth
news:dbr79q$ftg$1...@news.sap-ag.de...
However, a path as a sequence of characters has even less
meaning - I can't think of a use, while I have an application
where traversing a path as a sequence of path elements makes
perfect sense: I need to descend the directory structure, directory
by directory, looking for specific files and types.
John Roth
>
> Daniel
Duncan Booth
> Here's some code I just wrote seconds ago to construct a path for a
> scp upload:
>
> """
> DST_DIRPATH = path("host:~/destination")
> RSS_EXT = "rss"
>
> dst_filenamebase = os.extsep.join([postcode.lower(), RSS_EXT])
> dst_filepath = DST_DIRPATH.joinpath(dst_filenamebase)
> """
>
> With the current path implementation, this Just Works.
It isn't at all obvious to me that it works:
>>> import os
>>> from path import path
>>> postcode = "AA2 9ZZ"
>>> DST_DIRPATH = path("host:~/destination")
>>> RSS_EXT = "rss"
>>>
>>> dst_filenamebase = os.extsep.join([postcode.lower(), RSS_EXT])
>>> dst_filepath = DST_DIRPATH.joinpath(dst_filenamebase)
>>> print dst_filepath
host:~/destination\aa2 9zz.rss
> If I were using
> something that parsed and understood paths, the scp/rcp convention of
> host:filename would either cause an error or have to be programmed in
> separately. The current implementation is much more flexible.
You still have to program your scp path separately from your filesystem
path in order to handle the different conventions for path separator
characters and maybe also escaping special characters in the path (I don't
use scp much so I don't know if this is required).
> What are the practical advantages and conveniences of *not*
> subclassing from basestring?
Simplification of the api: not having methods such as center, expandtabs
and zfill.
Not having the base class change from str to unicode depending on which
system you run your code?
Fewer undetected bugs (explicit is better than implicit)?
Perhaps none of these matter in practice. As I said elsewhere I haven't
used path for anything real, so I'm still finding surprises such as why
this doesn't do what I expect:
>>> p = path('a/b')
>>> q = path('c/d')
>>> p+q
path(u'a/bc/d')
If path didn't subclass string then either this would have been
implemented, and probably would Do The Right Thing, or it wouldn't be
implemented so I'd quickly realise I needed to do something else. Instead
it does something suprising.
Andrew Dalke
> Personally I think the concept of a specific path type is a good one, but
> subclassing string just cries out to me as the wrong thing to do.
I disagree. I've tried using a class which wasn't derived from
a basestring and kept running into places where it didn't work well.
For example, "open" and "mkdir" take strings as input. There is no
automatic coercion.
>>> class Spam:
... def __getattr__(self, name):
... print "Want", repr(name)
... raise AttributeError, name
...
>>> open(Spam())
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: coercing to Unicode: need string or buffer, instance found
>>> import os
>>> os.mkdir(Spam())
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: coercing to Unicode: need string or buffer, instance found
>>>
The solutions to this are:
1) make the path object be derived from str or unicode. Doing
this does not conflict with any OO design practice (eg, Liskov
substitution).
2) develop a new "I represent a filename" protocol, probably done
via adapt().
I've considered the second of these but I think it's a more
complicated solution and it won't fit well with existing APIs
which do things like
if isinstance(input, basestring):
input = open(input, "rU")
for line in input:
print line
I showed several places in the stdlib and in 3rd party packages
where this is used.
> In other words, to me a path represents something in a filesystem,
Being picky - or something that could be in a filesystem.
> the fact that it
> has one, or indeed several string representations does not mean that the
> path itself is simply a more specific type of string.
I didn't follow this.
> You should need an explicit call to convert a path to a string and that
> forces you when passing the path to something that requires a string to
> think whether you wanted the string relative, absolute, UNC, uri etc.
You are broadening the definition of a file path to include URIs?
That's making life more complicated. Eg, the rules for joining
file paths may be different than the rules for joining URIs.
Consider if I have a file named "mail:da...@example.com" and I
join that with "file://home/dalke/badfiles/".
Additionally, the actions done on URIs are different than on file
paths. What should os.listdir("http://www.python.org/") do?
As I mentioned, I tried some classes which emulated file
paths. One was something like
class TempDir:
"""removes the directory when the refcount goes to 0"""
def __init__(self):
self.filename = ... use a function from the tempfile module
def __del__(self):
if os.path.exists(self.filename):
shutil.rmtree(self.filename)
def __str__(self):
return self.filename
I could do
dirname = TempDir()
but then instead of
os.mkdir(dirname)
tmpfile = os.path.join(dirname, "blah.txt")
I needed to write it as
os.mkdir(str(dirname))
tmpfile = os.path.join(str(dirname), "blah.txt"))
or have two variables, one which could delete the
directory and the other for the name. I didn't think
that was good design.
If I had derived from str/unicode then things would
have been cleaner.
Please note, btw, that some filesystems are unicode
based and others are not. As I recall, one nice thing
about the path module is that it chooses the appropriate
base class at import time. My "str()" example above
does not and would fail on a Unicode filesystem aware
Python build.
> It may even be that we need a hierarchy of path
> classes: URLs need similar but not identical manipulations
> to file paths, so if we want to address the failings
> of os.path perhaps we should also look at the failings
> of urlparse at the same time.
I've found that hierarchies are rarely useful compared
to the number of times they are proposed and used. One
of the joys to me of Python is its deemphasis of class
hierarchies.
I think the same is true here. File paths and URIs are
sufficiently different that there are only a few bits
of commonality between them. Consider 'split' which
for files creates (dirname, filename) while for urls
it creates (scheme, netloc, path, query, fragment)
Andrew
da...@dalkescientific.com
Andrew Dalke
> You're right, conceptually a path
> HAS_A string description, not IS_A string, so from a pure OO point of
> view, it should not inherit string.
How did you decide it's "has-a" vs. "is-a"?
All C calls use a "char *" for filenames and paths,
meaning the C model file for the filesystem says
paths are strings.
Paths as strings fit the Liskov substitution principle
in that any path object can be used any time a
string is used (eg, "loading from " + filename)
Good information hiding suggests that a better API
is one that requires less knowledge. I haven't
seen an example of how deriving from (unicode)
string makes things more complicated than not doing so.
Andrew
da...@dalkescientific.com
Scott David Daniels
> BTW, does it matter at all in practical use that the base class of path
> varies between str and unicode depending on the platform?
Isn't it even worse than this?
On Win2K & XP, don't the file systems have something to do with the
encoding? So D: (a FAT drive) might naturally be str, while C:
(an NTFS drive) might naturally be unicode. Even worse, would be a
path that switches in the middle (which it might do if we get to a
ZIP file or use the newer dir-in-file file systems.
--Scott David Daniels
Scott....@Acm.Org
Michael Hoffman
> However, a path as a sequence of characters has even less
> meaning - I can't think of a use, while I have an application
> where traversing a path as a sequence of path elements makes
> perfect sense: I need to descend the directory structure, directory
> by directory, looking for specific files and types.
I *have* used a path as a sequence of characters before. I had to deal
with a bunch of filenames that were formatted like "file_02832.a.txt"
I can see the case for a path as a sequence of elements, although in
practice, drive letters, extensions, and alternate streams complicate
things.
But as the discussion here unfolds I'm starting to feel that the
advantages of using a possibly more "meaningful" approach to path as a
sequence of elements are overwhelmed by the practical advantages of
using a basestring. Mainly, that you can use it anywhere a basestring
would be used today and it Just Works.
--
Michael Hoffman
Terry Reedy
"Daniel Dittmar" <daniel....@sap.corp> wrote in message
news:dbr79q$ftg$1...@news.sap-ag.de...
> Duncan Booth wrote:
>> I would have expected a
>> path object to be a sequence of path elements rather than a sequence of
>> characters.
Glad I'm not the only oddball.
> Maybe it's nitpicking, but I don't think that a path object should be a
> 'sequence of path elements' in an iterator context.
>
> This means that
>
> for element in pathobject:
>
> has no intuitive meaning for me, so it shouldn't be allowed.
???? The internal equivalent of (simplified, omitting error checking,
etc.)
for dir in pathobject:
if isdir(dir): cd(dir)
*is*, in essence, what the OS mainly does with paths (after splitting the
string representation into pieces).
Directory walks also work with paths as sequences (stacks, in particular).
Terry J. Reedy
Peter Hansen
Peter Hansen
> BTW, does it matter at all in practical use that the base class of path
> varies between str and unicode depending on the platform?
I haven't seen any problem. I confess I can't even imagine exactly what
the problem might be, since they're both subclasses of basestring,
aren't they?
And current code should have exactly the same issues when using str or
unicode in all the calls that path() merely wraps.
So does it matter in practical use when one faces this issue and is
*not* using "path"?
-Peter
Peter Hansen
> As I said elsewhere I haven't
> used path for anything real, so I'm still finding surprises such as why
> this doesn't do what I expect:
>
>>>>p = path('a/b')
>>>>q = path('c/d')
>>>>p+q
>
> path(u'a/bc/d')
Just a note, though you probably know, that this is intended to be
written this way with path:
>>> p / q
path(u'a/b/c/d')
-Peter
Peter Hansen
> FYI: I modified the path module a bit so that it fits many of the suggestions
> from python-dev, and put the result in the Python CVS tree under
> nondist/sandbox/path.
By the way, thanks for doing this Reinhold!
> Most prominent change is that it doesn't inherit from str/unicode anymore.
> I found this distinction important, because as a str subclass the Path object
> has many methods that don't make sense for it.
On this topic, has anyone ask the original author (Jason Orendorff)
whether he has some background on this decision that might benefit the
discussion? Given the elegance of the design of the path module, I
would think if he has an opinion on the matter it is probably based on
more thought than any of us have given it so far.
And maybe he would even say that it was a wrong decision at the time and
he'd do it differently the next time.
-Peter
Oliver Andrich
2005/7/22, Michael Hoffman <cam....@mh391.invalid>:
> What is this Java Path class? I have been STFWing and have found nothing
> on it in the. Indeed if you search for "Java Path class" (with quotes)
> almost half of the pages are this message from Guido. ;)
>
> Any Java hackers here want to tell us of the wonders of the Java Path
> class? I would be interested in seeing how other OO languages deal with
> paths.
I guess the nearest Java class comparable with path is the File class.
http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html
And as I am a so called Java hacker, I highly appreciate path as a
module for my python projects and in my eyes it is the natural way to
address files/paths. At least it is more natural to me then the os,
os.path, etc. pp. bundle, that has grown over the time.
I would love to see path inside Python's stdlib.
Best regards,
Oliver
--
Oliver Andrich <oliver....@gmail.com> --- http://fitheach.de/
Terry Reedy
"Michael Hoffman" <cam....@mh391.invalid> wrote in message
news:dbqb5p$f5e$1...@gemini.csx.cam.ac.uk...
>> on 22.07.2005 00:21 Michael Hoffman said the following:
>>> Any Java hackers here want to tell us of the wonders of the Java Path
>>> class?
>
> http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html
Very interesting. Java's File class is a system-independent "abstract
representation of file and directory pathnames" which is constructed from
and converted to system-dependent string-form pathnames (including URL/URI
file:... forms). A File consist of an optional prefix and a *sequence* of
zero or more string names.
In other words, Java's File class is what Duncan and I thought Python's
Path class might or possibly should be. So this internal representation
might be worth considering as an option. Of course, if the interface is
done right, it mostly should not make too much difference to the user.
Terry J. Reedy
Michael Hoffman
> Duncan Booth wrote:
>
>> BTW, does it matter at all in practical use that the base class of
>> path varies between str and unicode depending on the platform?
>
> Isn't it even worse than this?
> On Win2K & XP, don't the file systems have something to do with the
> encoding? So D: (a FAT drive) might naturally be str, while C:
> (an NTFS drive) might naturally be unicode.
The current path module handles these situations at least as well as the
libraries that come with Python do. ;-)
--
Michael Hoffman
Michael Hoffman
> When files are opened through a "path" object -- e.g.
> path('name').open() -- then file.name returns the path object that was
> used to open it.
Also works if you use file(path('name')) or open(path('name')).
--
Michael Hoffman
Reinhold Birkenfeld
> Duncan Booth wrote:
>> Personally I think the concept of a specific path type is a good one, but
>> subclassing string just cries out to me as the wrong thing to do.
>
> I disagree. I've tried using a class which wasn't derived from
> a basestring and kept running into places where it didn't work well.
> For example, "open" and "mkdir" take strings as input. There is no
> automatic coercion.
Well, as a Path object provides both open() and mkdir() functions, these use
cases are covered. And that's the point of the Path class: Every common use
you may have for a path is implemented as a method.
So, it's maybe a good thing that for uncommon uses you have to explicitly
"cast" the path to a string.
That's a valid point. However, if Path is not introduced as a string,
most new users will not try to use a Path instance where a string is
needed, just as you wouldn't try to pass a list where a string is wanted.
>> You should need an explicit call to convert a path to a string and that
>> forces you when passing the path to something that requires a string to
>> think whether you wanted the string relative, absolute, UNC, uri etc.
>
> You are broadening the definition of a file path to include URIs?
> That's making life more complicated. Eg, the rules for joining
> file paths may be different than the rules for joining URIs.
> Consider if I have a file named "mail:da...@example.com" and I
> join that with "file://home/dalke/badfiles/".
>
> Additionally, the actions done on URIs are different than on file
> paths. What should os.listdir("http://www.python.org/") do?
I agree. Path is only for local filesystem paths (well, in UNIX they could
as well be remote, but that's thanks to the abstraction the filesystem
provides, not Python).
> As I mentioned, I tried some classes which emulated file
> paths. One was something like
>
> class TempDir:
> """removes the directory when the refcount goes to 0"""
> def __init__(self):
> self.filename = ... use a function from the tempfile module
> def __del__(self):
> if os.path.exists(self.filename):
> shutil.rmtree(self.filename)
> def __str__(self):
> return self.filename
>
> I could do
>
> dirname = TempDir()
>
> but then instead of
>
> os.mkdir(dirname)
> tmpfile = os.path.join(dirname, "blah.txt")
>
> I needed to write it as
>
> os.mkdir(str(dirname))
> tmpfile = os.path.join(str(dirname), "blah.txt"))
>
> or have two variables, one which could delete the
> directory and the other for the name. I didn't think
> that was good design.
I can't follow. That's clearly not a Path but a custom object of yours.
However, I would have done it differently: provide a "name" property
for the object, and don't call the variable "dirname", which is confusing.
> If I had derived from str/unicode then things would
> have been cleaner.
>
> Please note, btw, that some filesystems are unicode
> based and others are not. As I recall, one nice thing
> about the path module is that it chooses the appropriate
> base class at import time. My "str()" example above
> does not and would fail on a Unicode filesystem aware
> Python build.
There's no difference. The only points where the type of a Path
object' underlying string is decided are Path.cwd() and the
Path constructor.
Reinhold
Michael Hoffman
> Andrew Dalke wrote:
>
>>I disagree. I've tried using a class which wasn't derived from
>>a basestring and kept running into places where it didn't work well.
>>For example, "open" and "mkdir" take strings as input. There is no
>>automatic coercion.
>
> Well, as a Path object provides both open() and mkdir() functions, these use
> cases are covered. And that's the point of the Path class: Every common use
> you may have for a path is implemented as a method.
Except when you pass the path to a function written by someone else
> So, it's maybe a good thing that for uncommon uses you have to explicitly
> "cast" the path to a string.
Where uncommon uses include passing the path object to any code you
don't control? The stdlib can be fixed, this other stuff can't.
>>The solutions to this are:
>> 1) make the path object be derived from str or unicode. Doing
>>this does not conflict with any OO design practice (eg, Liskov
>>substitution).
>>
>> 2) develop a new "I represent a filename" protocol, probably done
>>via adapt().
>>
>>I've considered the second of these but I think it's a more
>>complicated solution and it won't fit well with existing APIs
>>which do things like
>>
>>
>> if isinstance(input, basestring):
>> input = open(input, "rU")
>> for line in input:
>> print line
>>
>>I showed several places in the stdlib and in 3rd party packages
>>where this is used.
>
> That's a valid point. However, if Path is not introduced as a string,
> most new users will not try to use a Path instance where a string is
> needed, just as you wouldn't try to pass a list where a string is wanted.
But many functions were written expecting lists as arguments but also
work for strings, and do not require an explicit list(mystring) before
calling the function.
--
Michael Hoffman
Michael Hoffman
>> Most prominent change is that it doesn't inherit from str/unicode
>> anymore.
>> I found this distinction important, because as a str subclass the Path
>> object
>> has many methods that don't make sense for it.
>
> On this topic, has anyone ask the original author (Jason Orendorff)
> whether he has some background on this decision that might benefit the
> discussion?
My impression is that he doesn't have a lot of spare cycles for this. He
didn't have anything to add to the python-dev discussion when I informed
him of it. I'd love to hear what he had to say about the design.
--
Michael Hoffman
John Machin
Try this:
A file-system is a maze of twisty little passages, all alike. Junction
== directory. Cul-de-sac == file. Fortunately it is signposted. You are
dropped off at one of the entrance points ("current directory", say).
You are given a route (a "path") to your destination. The route consists
of a list of intermediate destinations.
for element in pathobject:
follow_sign_post_to(element)
Exception-handling strategy: Don't forget to pack a big ball of string.
Anecdotal evidence is that breadcrumbs are unreliable.
Cheers,
John
John Machin
> John Roth wrote:
>
>> However, a path as a sequence of characters has even less
>> meaning - I can't think of a use, while I have an application
>> where traversing a path as a sequence of path elements makes
>> perfect sense: I need to descend the directory structure, directory
>> by directory, looking for specific files and types.
>
>
> I *have* used a path as a sequence of characters before. I had to deal
> with a bunch of filenames that were formatted like "file_02832.a.txt"
Ya, ya , ya, only two days ago I had to lash up a quick script to find
all files matching r"\d{8,8}[A-Za-z]{0,3}$" and check that the expected
number were present in each sub-directory (don't ask!).
BUT you are NOT using "a path as a sequence of characters". Your
filename is a path consisting of one element. The *element* is an
instance of basestring, to which you can apply all the string methods
and the re module.
Peter Hansen
Since that's exactly what the path module does, it's not surprising.
Practically everything that path does, with a few useful exceptions, is
a thin wrapper around the existing calls. path.open, for example is
merely this:
def open(self, mode='r'):
return file(self, mode)
-Peter
George Sakkis
> George Sakkis wrote:
> > You're right, conceptually a path
> > HAS_A string description, not IS_A string, so from a pure OO point of
> > view, it should not inherit string.
>
> How did you decide it's "has-a" vs. "is-a"?
>
> All C calls use a "char *" for filenames and paths,
> meaning the C model file for the filesystem says
> paths are strings.
Bringing up how C models files (or anything else other than primitive types for that matter) is not
a particularly strong argument in a discussion on OO design ;-)
> Paths as strings fit the Liskov substitution principle
> in that any path object can be used any time a
> string is used (eg, "loading from " + filename)
Liskov substitution principle imposes a rather weak constraint on when inheritance should not be
used, i.e. it is a necessary condition, but not sufficient. Take for example the case where a
PhoneNumber class is subclass of int. According to LSP, it is perfectly ok to add phone numbers
together, subtract them, etc, but the result, even if it's a valid phone number, just doesn't make
sense.
> Good information hiding suggests that a better API
> is one that requires less knowledge. I haven't
> seen an example of how deriving from (unicode)
> string makes things more complicated than not doing so.
I wouldn't say more complicated, but perhaps less intuitive in a few cases, e.g.:
> path(r'C:\Documents and Settings\Guest\Local Settings').split()
['C:\\Documents', 'and', 'Settings\\Guest\\Local', 'Settings']
instead of
['C:', 'Documents and Settings', 'Guest', 'Local Settings']
I just noted that conceptually a path is a composite object consisting of many properties (dirname,
extension, etc.) and its string representation is just one of them. Still, I'm not suggesting that a
'pure' solution is better that a more practical that covers most usual cases.
George
Michael Hoffman
> Practically everything that path does, with a few useful exceptions, is
> a thin wrapper around the existing calls.
If the implementation is easy to explain, it may be a good idea.
OT: I just realized you can now type in "python -m this" at the command
line, which is convenient, but strange.
--
Michael Hoffman
Neil Hodgson
> Isn't it even worse than this?
> On Win2K & XP, don't the file systems have something to do with the
> encoding? So D: (a FAT drive) might naturally be str, while C:
> (an NTFS drive) might naturally be unicode.
This is generally safe as Windows is using unicode internally and
provides full-fidelity access to the FAT drive using unicode strings.
You can produce failures if you try to create files with names that can
not be represented but you would see a similar failure with byte string
access.
> Even worse, would be a
> path that switches in the middle (which it might do if we get to a
> ZIP file or use the newer dir-in-file file systems.
If you are promoting from byte strings with a known encoding to
unicode path objects then this should always work.
Neil
Andrew Dalke
> Bringing up how C models files (or anything else other than primitive types
> for that matter) is not a particularly strong argument in a discussion on
> OO design ;-)
While I have worked with C libraries which had a well-developed
OO-like interface, I take your point.
Still, I think that the C model of a file system should be a
good fit since after all C and Unix were developed hand-in-hand. If
there wasn't a good match then some of the C path APIs should be
confusing or complicated. Since I don't see that it suggests that
the "path is-a string" is at least reasonable.
> Liskov substitution principle imposes a rather weak constraint
Agreed. I used that as an example of the direction I wanted to
go. What principles guide your intuition of what is a "is-a"
vs a "has-a"?
> Take for example the case where a PhoneNumber class is subclass
> of int. According to LSP, it is perfectly ok to add phone numbers
> together, subtract them, etc, but the result, even if it's a valid
> phone number, just doesn't make sense.
Mmm, I don't think an integer is a good model of a phone number.
For example, in the US
00148762040828
will ring a mobile number in Sweden while
148762040828
will give a "this isn't a valid phone number" message.
Yet both have the same base-10 representation. (I'm not using
a syntax where leading '0' indicates an octal number. :)
> I wouldn't say more complicated, but perhaps less intuitive in a few cases, e.g.:
>
>> path(r'C:\Documents and Settings\Guest\Local Settings').split()
> ['C:\\Documents', 'and', 'Settings\\Guest\\Local', 'Settings']
> instead of
> ['C:', 'Documents and Settings', 'Guest', 'Local Settings']
That is why the path module using a different method to split
on pathsep vs. whitespace. I get what you are saying, I just think
it's roughly equivalent to appealing to LSP in terms of weight.
Mmm, then there's a question of the usefulness of ".lower()" and
".expandtabs()" and similar methods. Hmmm....
> I just noted that conceptually a path is a composite object consisting of
> many properties (dirname, extension, etc.) and its string representation
> is just one of them. Still, I'm not suggesting that a 'pure' solution is
> better that a more practical that covers most usual cases.
For some reason I think that
path.dirname()
is better than
path.dirname
Python has properties now so the implementation of the latter is
trivial - put a @property on the line before the "def dirname(self):".
I think that the string representation of a path is so important that
it *is* the path. The other things you call properties aren't quite
properties in my model of a path and are more like computable values.
I trust my intuition on this, I just don't know how to justify it, or
correct it if I'm wrong.
Andrew
da...@dalkescientific.com
George Sakkis
> [snipped]
>
> > Take for example the case where a PhoneNumber class is subclass
> > of int. According to LSP, it is perfectly ok to add phone numbers
> > together, subtract them, etc, but the result, even if it's a valid
> > phone number, just doesn't make sense.
>
> Mmm, I don't think an integer is a good model of a phone number.
> For example, in the US
> 00148762040828
> will ring a mobile number in Sweden while
> 148762040828
> will give a "this isn't a valid phone number" message.
That's why phone numbers would be a subset of integers, i.e. not every integer would correspond to a
valid number, but with the exception of numbers starting with zeros, all valid numbers would be an
integers. Regardless, this was not my point; the point was that adding two phone numbers or
subtracting them never makes sense semantically.
> [snipped]
>
> > I just noted that conceptually a path is a composite object consisting of
> > many properties (dirname, extension, etc.) and its string representation
> > is just one of them. Still, I'm not suggesting that a 'pure' solution is
> > better that a more practical that covers most usual cases.
>
> For some reason I think that
>
> path.dirname()
>
> is better than
>
> path.dirname
>
> Python has properties now so the implementation of the latter is
> trivial - put a @property on the line before the "def dirname(self):".
Sorry, I used the term 'property' in the broad sense, as the whole exposed API, not the specific
python feature; I've no strong preference between path.dirname and path.dirname().
> I think that the string representation of a path is so important that
> it *is* the path.
There are (at least) two frequently used path string representations, the absolute and the relative
to the working directory. Which one *is* the path ? Depending on the application, one of them woud
be more natural choice than the other.
> I trust my intuition on this, I just don't know how to justify it, or
> correct it if I'm wrong.
My intuition also happens to support subclassing string, but for practical reasons rather than
conceptual.
George
Andrew Dalke
> That's why phone numbers would be a subset of integers, i.e. not every
> integer would correspond to a valid number, but with the exception of
> numbers starting with zeros, all valid numbers would be an integers.
But it's that exception which violates the LSP.
With numbers, if x==y then (x,y) = (y,x) makes no difference.
If phone numbers are integers then 001... == 01... but swapping
those two numbers makes a difference. Hence they cannot be modeled
as integers.
> Regardless, this was not my point; the point was that adding
> two phone numbers or subtracting them never makes sense semantically.
I agree. But modeling them as integers doesn't make sense either.
Your example of adding phone numbers depends on them being represented
as integers. Since that representation doesn't work, it makes sense
that addition of phone number is suspect.
> There are (at least) two frequently used path string representations,
> the absolute and the relative to the working directory. Which one *is*
> the path ? Depending on the application, one of them woud be more
> natural choice than the other.
Both. I don't know why one is more natural than the other.
>> I trust my intuition on this, I just don't know how to justify it, or
>> correct it if I'm wrong.
>
> My intuition also happens to support subclassing string, but for
> practical reasons rather than conceptual.
As you may have read elsewhere in this thread, I give some examples
of why subclassing from string fits best with existing code.
Even if there was no code base, I think deriving from string is the
right approach. I have a hard time figuring out why though. I think
if the lowest level Python/C interface used a "get the filename"
interface then perhaps it wouldn't make a difference. Which means
I'm also more guided by practical reasons than conceptual.
Andrew
da...@dalkescientific.com
pa...@boddie.org.uk
> Reinhold Birkenfeld wrote:
> > Probably as Terry said: a path is both a list and a string.
[...]
> One way to divide this is solely based on path separators:
>
> ['c:', 'windows', 'system32:altstream', 'test.dir',
> 'myfile.txt.zip:altstream']
I would argue that any proposed solution has to work with VMS
pathnames. ;-)
> The current stdlib solution, os.path.splitext(os.path.splitext(filename)
> [0])[0] is extremely clunky, and I have long desired something better.
> (OK, using filename.split(os.extsep) works a little better, but you get
> the idea.)
And also with unusual (eg. RISC OS) filename extensions.
To do any justice to the existing solutions, any PEP should review at
least the following projects:
* The path module (of course):
http://www.jorendorff.com/articles/python/path/
* The py.path module (or at least the ideas for it):
http://codespeak.net/py/current/doc/future.html
* itools.uri
http://www.ikaaro.org/itools
* Results from the "Object-Oriented File System Virtualisation"
project in the "Summer of Code" programme:
http://wiki.python.org/moin/SummerOfCode
And I hope that the latter project is reviewing some of the other work,
if only to avoid the "framework proliferation" that people keep
complaining about.
Paul
Peter Hansen
> "Andrew Dalke" <da...@dalkescientific.com> wrote:
>>I think that the string representation of a path is so important that
>>it *is* the path.
>
> There are (at least) two frequently used path string representations,
> the absolute and the relative to the working directory. Which one
> *is* the path ? Depending on the application, one of them woud
> be more natural choice than the other.
Sorry, George, but that's now how it works.
Whether using the regular string-based Python paths or the new path
module, a path *is* either absolute or relative, but cannot be both at
the same time.
This is therefore not an issue of "representation" but one of state.
-Peter
Duncan Booth
> Just a note, though you probably know, that this is intended to be
> written this way with path:
>
> >>> p / q
> path(u'a/b/c/d')
I know, but it really doesn't look right to me.
I think that my fundamental problem with all of this is that by making path
a subclass of str/unicode it inherits inappropriate definitions of some
common operations, most obviously addition, iteration and subscripting.
These operations have obvious meaning for paths which is not the same as
the meaning for string. Therefore (in my opinion) the two ought to be
distinct.
Bengt Richter
<indulging what="my penchant for seeking the general behind the specific ;-)" >
ISTM a path is essentially a representation of a script whose interpretation
by an orderly choice of interpreters finally leads to accessing to some entity,
typically a serial data representation, through an object, perhaps a local proxy,
that has standard methods for accessing the utimate object's desired info.
IOW, a path sequence is like a script text that has been .splitline()'d and
and the whole sequence fed to a local interpreter, which might chew through multiple
lines on its own, or might invoke interpreters on another network to deal with the
rest of the script, or might use local interpreters for various different kinds of
access (e.g., after seeing 'c:' vs 'http://' vs '/c' vs '//c' etc. on the platform
defining the interpretation of the head element).
Turning a single path string into a complete sequence of elements is not generally possible
unless you have local knowledge of the syntax of the entire tail beyond the the prefix
you have to deal with. Therefore, a local platform-dependent Pathobject class should, I think,
only recognize prefixes that it knows how to process or delegate processing for, leaving
the interpretation of the tail to the next Pathobject instance, however selected and/or
located.
So say (this is just a sketch, mind ;-)
po = Pathobject(<string representation of whole path>)
results in a po that splits out (perhaps by regex) a prefix, a first separator/delimiter,
and the remaining tail. E.g., in class Pathobject,
def __init__(self, pathstring=None)
if pathstring is None: #do useful default??
self.pathstring = pathstring
self.prefix, self.sep, self.tail = self.splitter(pathstring)
if self.prefix in self.registered_prefixes:
self.child = self.registered_prefixes[self.prefix](self.tail)
else:
self.child = []
self.opened_obj = None
Then the loop inside a local pathobject's open method po.open()
might go something like
def open(self, *mode, **kw):
if self.child:
self.opened_obj = self.child.open(self.tail, *mode, **kw)
else:
self.opened_obj = file(self.pathstring, *mode)
return self
And closing would just go to the immediately apparent opened object, and
if that had complex closing to do, it would be its responsibility to deal
with itself and its child-derived objects.
def close(self):
self.opened_object.close()
The point is that a given pathobject could produce a new or modified pathobject child
which might be parsing urls instead of windows file system path strings or could
yield an access object producing something entirely synthetic.
A synthetic capability could easily be introduced if the local element pathobject
instance looked for e.g., 'synthetic://' as a possible first element (prefix) string representation,
and then passed the tail to a subclass defining synthetic:// path interpretation.
E.g., 'synthetic://temp_free_diskspace' could be a platform-independent way to get such info as that.
Opening 'testdata:// ...' might be an interesting way to feed test suites, if pathobject subclasses
could be registered locally and found via the head element's string representation.'
One point from this is that a path string represents an ordered sequence of elements, but is heterogenous,
and therefore has potentially heterogenous syntax reflected in string tails with syntax that should be
interpreted differently from the prefix syntax.
Each successive element of a path string effectively requires an interpreter for that stage of access
pursuit, and the chain of processing may result in different path entities/objects/representations
on different systems, with different interpretations going on, sharing only that they are part of the
process of getting access to something and providing access services, if it's not a one-shot access.
This would also be a potential way to create access to a foreign file system in pure python if desired,
so long as there was a way of accessing the raw data to build on, e.g. a raw stuffit floppy, or a raw
hard disk if there's the required privileges. Also 'zip://' or 'bzip2://' could be defined
and registered by a particular script or in an automatic startup script. 'encrypted://' might be interesting.
Or if polluting the top namespace was a problem, a general serialized data access header element
might work, e.g., 'py_sda://encrypted/...'
This is very H[ot]OTTOMH (though it reflects some thoughts I've had before, so be kind ;-)
For compatibility with the current way of doing things, you might want to do an automatic open
in the Pathobject constructor, but I don't really like that. It's easy enough to tack on ".open()"
po = Pathobject('py_sda://encrypted/...')
po.open() # plain read_only text default file open, apparently, but encrypted does binary behind the scenes
print po.read()
po.close()
Say, how about
if Pathobject('gui://message_box/yn/continue processing?').open().read().lower()!='y':
raise SystemExit, "Ok, really not continuing ;-)"
An appropriate registered subclass for the given platform, returned when the
Pathobject base class instantiates and looks at the first element on open() and delegates
would make that possible, and spelled platform-independently as in the code above.
</indulging>
Regards,
Bengt Richter
Daniel Dittmar
> However, a path as a sequence of characters has even less
> meaning - I can't think of a use, while I have an application
That's true. But the arguments for path objects as strings go more in
the direction of using existing functions that expect strings.
> where traversing a path as a sequence of path elements makes
> perfect sense: I need to descend the directory structure, directory
> by directory, looking for specific files and types.
But then your loop doesn't need the individual path elements, but rather
sub-path objects
for p in pathobj.stepdown ('/usr/local/bin'):
if p.join (searchedFile):
whatever
I'm not saying that there isn't any use for having a list of path
elements. But it isn't that common, so it should get an methodname to
make it more explicit.
Daniel
Daniel Dittmar
> for dir in pathobject:
> if isdir(dir): cd(dir)
>
> *is*, in essence, what the OS mainly does with paths (after splitting the
> string representation into pieces).
That's why there is rarely a need to to it in Python code.
> Directory walks also work with paths as sequences (stacks, in particular).
I'd say it works with stacks of pathes, not with stacks of path elements.
Ron Adam
> <indulging what="my penchant for seeking the general behind the specific ;-)" >
<fold>
>
> Say, how about
>
> if Pathobject('gui://message_box/yn/continue processing?').open().read().lower()!='y':
> raise SystemExit, "Ok, really not continuing ;-)"
>
> An appropriate registered subclass for the given platform, returned when the
> Pathobject base class instantiates and looks at the first element on open() and delegates
> would make that possible, and spelled platform-independently as in the code above.
I like it. ;-)
No reason why a path can't be associated to any tree based object.
> </indulging>
>
> Regards,
> Bengt Richter
<more indulging>
I wasn't sure what to comment on, but it does point out some interesting
possibilities I think.
A path could be associated to any file-like object in an external (or
internal) tree structure. I don't see any reason why not.
In the case of an internal file-like object, it could be a series of
keys in nested dictionaries. Why not use a path as a dictionary
interface?
So it sort of raises the question of how tightly a path object should be
associated to a data structure? When and where should the path object
determine what the final path form should be? And how smart should it
be as a potential file-like object?
Currently the device name is considered part of the path, but if instead
you treat the device as an object, it could open up more options.
(Which would extend the pattern of your example above. I think.)
(also a sketch.. so something like...)
# Initiate a device path object.
apath = device('C:').path(initial_path)
# Use it to get and put data
afile = apath.open(mode,position,units) # defaults ('r','line',next)
aline = afile.read().next() # read next unit, or as an iterator.
afile.write(line)
afile.close()
# Manually manipulate the path
apath.append('something') # add to end of path
apath.remove() # remove end of path
alist = apath.split() # convert to a list
apath.join(alist) # convert list to a path
astring = str(apath()) # get string from path
apath('astring') # set path to string
apath.validate() # make sure it's valid
# Iterate and navigate the path
apath.next() # iterate path objects
apath.next(search_string) # iterate with search string
apath.previous() # go back
apath.enter() # enter directory
apath.exit() # exit directory
# Close it when done.
apath.close()
etc...
With this you can iterate a file system as well as it's files. ;-)
(Add more or less methods as needed of course.)
apath = device(dev_obj).path(some_path_sting)
apath.open().write(data).close()
or if you like...
device(dev_obj).append(path_sting).open().write(data).close()
Just a few thoughts,
Cheers,
Ron
qvx
> Bengt Richter wrote:
>
> > <indulging what="my penchant for seeking the general behind the specific ;-)"
There is a thing called "Asynchronous pluggable protocol". It is
Microsoft's technology (don't flame me now):
"""
Asynchronous pluggable protocols enable developers to create pluggable
protocol handlers, MIME filters, and namespace handlers that work with
Microsoft® Internet Explorer...
Applications can use pluggable protocol handlers to handle a custom
Uniform Resource Locator (URL) protocol scheme or filter data for a
designated MIME type.
"""
In other words you can develop you own plugin which would allow
Internet Explorer to open URLs like "rar://c/my/doc/book.rar". (I was
going to write plugin for .rar in order to enable offsite browsing of
downloaded portions of web sites, all from an archive file).
You could give it a look. If only to see that it is Mycrosofthonic:
http://msdn.microsoft.com/workshop/networking/pluggable/overview/aplugprot_overviews_entry.asp.
Qvx