windows and file names > 256 bytes

[Albert-Jan Roskam]

Hi,

Consider the following calls, where very_long_path is more than 256 bytes:
[1] os.mkdir(very_long_path)
[2] os.getsize(very_long_path)
[3] shutil.rmtree(very_long_path)

I am using Python 2.7 and [1] and [2] fail under Windows XP [3] fails
under Win7 (not sure about XP). This is even when I use the "special"
notations \\?\c:\dir\file or \\?\UNC\server\share\file, e.g.
os.path.getsize("\\\\?\\" + "c:\\dir\\file")
(Oddly, os.path.getsize(os.path.join("\\\\?", "c:\\dir\\file")) will
truncate the prefix)

My questions:
1. How can I get the file size of very long paths under XP?
2. Is this a bug in Python? I would prefer if Python dealt with the gory
details of Windows' silly behavior.

Regards,
Albert-Jan

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

[Albert-Jan Roskam]

Hi,

Consider the following calls, where very_long_path is more than 256 bytes:
[1] os.mkdir(very_long_path)
[2] os.getsize(very_long_path)
[3] shutil.rmtree(very_long_path)

I am using Python 2.7 and [1] and [2] fail under Windows XP [3] fails

under Win7 (not sure about XP). It throws: “WindowsError: [Error 206] The
filename or extension is too long” This is even when I use the "special"

[Steven D'Aprano]

On Thursday 25 June 2015 18:00, Albert-Jan Roskam wrote:

> Hi,
>
> Consider the following calls, where very_long_path is more than 256 bytes:
> [1] os.mkdir(very_long_path)
> [2] os.getsize(very_long_path)
> [3] shutil.rmtree(very_long_path)
>
> I am using Python 2.7 and [1] and [2] fail under Windows XP [3] fails
> under Win7 (not sure about XP). It throws: “WindowsError: [Error 206] The
> filename or extension is too long”

I don't think this is a bug. It seems to be a limitation of Windows.

https://msdn.microsoft.com/en-
us/library/windows/desktop/aa365247%28v=vs.85%29.aspx#maxpath

> This is even when I use the "special"
> notations \\?\c:\dir\file or \\?\UNC\server\share\file, e.g.
> os.path.getsize("\\\\?\\" + "c:\\dir\\file")

However, that may be a bug.

What happens if you use a Unicode string?

path = u"\\\\?\\c:a\\very\\long\\path"
os.mkdir(path)


Can you open an existing file?

open(u"\\\\?\\c:a\\very\\long\\path\\file.txt")

> (Oddly, os.path.getsize(os.path.join("\\\\?", "c:\\dir\\file")) will
> truncate the prefix)

That's worth reporting as a bug.

> My questions:
> 1. How can I get the file size of very long paths under XP?

If all else fails:

last = os.getcwd()
try:
os.chdir('C:/a/very/long')
os.chdir('path/with/many')
os.chdir('nested/folders')
os.path.getsize('/and/even/more/file.txt')
finally:
os.chdir(last)

> 2. Is this a bug in Python? I would prefer if Python dealt with the gory
> details of Windows' silly behavior.

I would say that it is a bug that it doesn't work with extended-length paths
(those starting with \\?\) but may or may not be a bug with regular paths.


--
Steve

[Chris Angelico]

On Thu, Jun 25, 2015 at 7:16 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
>> 2. Is this a bug in Python? I would prefer if Python dealt with the gory
>> details of Windows' silly behavior.
>
> I would say that it is a bug that it doesn't work with extended-length paths
> (those starting with \\?\) but may or may not be a bug with regular paths.

I'd go further and say that the OP is right in expecting Python to
deal with the gory details. Would it break anything for Python to
prepend \\?\ to all file names before giving them to certain APIs?
Then the current behaviour of stripping off that prefix would be fine.

Are there any times when you *don't* want Windows to use the
extended-length path?

ChrisA

[Tim Golden]

On 25/06/2015 10:23, Chris Angelico wrote:
> On Thu, Jun 25, 2015 at 7:16 PM, Steven D'Aprano
> <steve+comp....@pearwood.info> wrote:

>>> 2. Is this a bug in Python? I would prefer if Python dealt with the gory
>>> details of Windows' silly behavior.
>>
>> I would say that it is a bug that it doesn't work with extended-length paths
>> (those starting with \\?\) but may or may not be a bug with regular paths.
>

> I'd go further and say that the OP is right in expecting Python to
> deal with the gory details. Would it break anything for Python to
> prepend \\?\ to all file names before giving them to certain APIs?
> Then the current behaviour of stripping off that prefix would be fine.
>
> Are there any times when you *don't* want Windows to use the
> extended-length path?

Yes: when you're passing a relative filepath. Which could pretty much be
any time. As you might imagine, this has come up before -- there's an
issue on the tracker for it somewhere. I just don't think it's simple
enough for Python to know when and when not to use the extended path
syntax without danger of breaking something.

Bear in mind that the \\?\ prefix doesn't just extend the length: it
also allows otherwise special-cased characters such as "." or "..". It's
a general-purpose mechanism for handing something straight to the file
system without parsing it first.

TJG

[Chris Angelico]

On Thu, Jun 25, 2015 at 8:10 PM, Tim Golden <ma...@timgolden.me.uk> wrote:
>> Are there any times when you *don't* want Windows to use the
>> extended-length path?
>
> Yes: when you're passing a relative filepath. Which could pretty much be
> any time. As you might imagine, this has come up before -- there's an
> issue on the tracker for it somewhere. I just don't think it's simple
> enough for Python to know when and when not to use the extended path
> syntax without danger of breaking something.

Oh blah. So I suppose that means there's fundamentally no way to use a
long (>256 byte) relative path on Windows?

> Bear in mind that the \\?\ prefix doesn't just extend the length: it
> also allows otherwise special-cased characters such as "." or "..". It's
> a general-purpose mechanism for handing something straight to the file
> system without parsing it first.

Ohh. So... hmm. So what this really means is that a path could get
\\?\ prepended when, and ONLY when, it becomes absolute. Windows can
be a real pest...

ChrisA

[Mark Lawrence]

On 25/06/2015 09:00, Albert-Jan Roskam wrote:
> Hi,
>
> Consider the following calls, where very_long_path is more than 256 bytes:
> [1] os.mkdir(very_long_path)
> [2] os.getsize(very_long_path)
> [3] shutil.rmtree(very_long_path)
>
> I am using Python 2.7 and [1] and [2] fail under Windows XP [3] fails
> under Win7 (not sure about XP). It throws: “WindowsError: [Error 206] The

> filename or extension is too long” This is even when I use the "special"

> notations \\?\c:\dir\file or \\?\UNC\server\share\file, e.g.
> os.path.getsize("\\\\?\\" + "c:\\dir\\file")

> (Oddly, os.path.getsize(os.path.join("\\\\?", "c:\\dir\\file")) will
> truncate the prefix)
>

> My questions:
> 1. How can I get the file size of very long paths under XP?

Please see
https://msdn.microsoft.com/en-gb/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath

> 2. Is this a bug in Python? I would prefer if Python dealt with the gory
> details of Windows' silly behavior.

I don't see why Python should work around any particular limitation of
any given OS.

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[Joonas Liik]

It sounds to me more like it is possible to use long file names on windows
but it is a pain and in python, on windows it is basically impossible.

So shouldn't it be possible to maniulate these files with extended names..

I mean even if you had to use some special function to ask for long names
it would still be better than no support at all.

[Tim Golden]

On 25/06/2015 13:04, Joonas Liik wrote:
> It sounds to me more like it is possible to use long file names on windows
> but it is a pain and in python, on windows it is basically impossible.

Certainly not impossible: you could write your own wrapper function:

def extended_path(p):
return r"\\?\%s" % os.path.abspath(p)

where you knew that there was a possibility of long paths and that an
absolute path would work.

TJG

[Chris Angelico]

On Thu, Jun 25, 2015 at 9:06 PM, Mark Lawrence <bream...@yahoo.co.uk> wrote:
>> 2. Is this a bug in Python? I would prefer if Python dealt with the gory
>> details of Windows' silly behavior.
>
>
> I don't see why Python should work around any particular limitation of any
> given OS.

Check out the multiprocessing module, and then tell me whether it's
better that Python paper over the OS differences or if you'd rather do
all that yourself. The biggest difference left between Windows and
POSIX is that on Windows, your main module has to be importable (which
doesn't hurt on POSIX). Python deals with all the mess of "can we
fork, or do we have to do it differently?".

ChrisA

[Michael Torrie]

The OP mentions that even when he manually supplies extended paths,
os.mkdir, os.getsize, and shutil.rmtree return errors for him in Python
2.7. So there's more to this problem.

[Tim Golden]

He's probably not passing unicode strings: the extended path only works
for unicode string. For 3.x that's what you do by default.

TJG

[Terry Reedy]

If possible, please try the same operations with Python 3.4 or .5 before
making a report


--
Terry Jan Reedy

[rand...@fastmail.us]

On Thu, Jun 25, 2015, at 09:35, Michael Torrie wrote:
> The OP mentions that even when he manually supplies extended paths,
> os.mkdir, os.getsize, and shutil.rmtree return errors for him in Python
> 2.7. So there's more to this problem.

The byte versions of the underlying OS APIs use a 256-character buffer
to do conversion - he needs to also be passing unicode strings.

[Albert-Jan Roskam]

On Thu, 25 Jun 2015 14:37:55 +0100, Tim Golden wrote:

> On 25/06/2015 14:35, Michael Torrie wrote:
>> On 06/25/2015 06:34 AM, Tim Golden wrote:
>>> On 25/06/2015 13:04, Joonas Liik wrote:
>>>> It sounds to me more like it is possible to use long file names on
>>>> windows but it is a pain and in python, on windows it is basically
>>>> impossible.
>>>
>>> Certainly not impossible: you could write your own wrapper function:
>>>
>>> def extended_path(p):
>>> return r"\\?\%s" % os.path.abspath(p)
>>>
>>> where you knew that there was a possibility of long paths and that an
>>> absolute path would work.
>>

>> The OP mentions that even when he manually supplies extended paths,
>> os.mkdir, os.getsize, and shutil.rmtree return errors for him in Python
>> 2.7. So there's more to this problem.
>>
>>

> He's probably not passing unicode strings: the extended path only works
> for unicode string. For 3.x that's what you do by default.

Hi all,

Thanks for your replies. I've been messing with this a bit more. I
created a little test script (see below). However, this only works with
drive letters, not with UNC paths. I tried using os.chdir, DOS pushd,
subst, net use but they all don't seem to work with with looooong paths.
I finally managed to remove an absurdly long dir with shutil.rmtree,
after changing sys.setrecursionlimit. But my main goal was to get the
file size (and, actually, also the file owner) of a long file name on XP.

import os
import shutil
import sys

# create an insanely long directory tree
p = os.getenv("TEMP")
#p = ur"\\server\share\blah\temp"
tmpdir = p
os.chdir(tmpdir)
for i in xrange(1000):
tmpdir = os.path.join(tmpdir, "sub")
os.mkdir("\\\\?\\" + tmpdir)
#os.mkdir(u"\\\\?\\UNC" + tmpdir[1:])

# write a file to it
deep = "\\\\?\\" + os.path.join(tmpdir, "deep.txt")
assert os.path.exists(deep)
with open(deep, "w") as f:
f.write("Deep!\r\n")

# try if the file size can be determined (requires special \\?\ notation)
print "@@@@ %d bytes" % os.path.getsize(deep)

# now delete the whole directory and its contents.
path = "\\\\?\\" + os.path.join(p, "sub")
path = path.decode(sys.getfilesystemencoding())
sys.setrecursionlimit(10 ** 7) # net use, pushd, subst will not work
shutil.rmtree(path)

Any feedback is welcome. I will post the solution somewhere so somebody
else will be spared this nuisance. :-)

[Albert-Jan Roskam]

<snip>

> import os import shutil import sys
>
> # create an insanely long directory tree p = os.getenv("TEMP")
> #p = ur"\\server\share\blah\temp"
> tmpdir = p os.chdir(tmpdir)
> for i in xrange(1000):
> tmpdir = os.path.join(tmpdir, "sub") os.mkdir("\\\\?\\" + tmpdir)
> #os.mkdir(u"\\\\?\\UNC" + tmpdir[1:])
>
> # write a file to it deep = "\\\\?\\" + os.path.join(tmpdir, "deep.txt")
> assert os.path.exists(deep)

sorry, this "assert" should of course follow 'with open(..'

[Tobiah]

On 06/24/2015 10:45 AM, Albert-Jan Roskam wrote:
> Hi,
>
> Consider the following calls, where very_long_path is more than 256 bytes:
> [1] os.mkdir(very_long_path)
> [2] os.getsize(very_long_path)
> [3] shutil.rmtree(very_long_path)
>
> I am using Python 2.7 and [1] and [2] fail under Windows XP [3] fails

the prefix)
>
> My questions:
> 1. How can I get the file size of very long paths under XP?

As a workaround, could you use multiple calls to os.chdir()
to get to where you need to do your operations, then use
relative paths from there?

Tobiah