recfile.py hits Windows filename length limit

[Kiran Subbaraman]

I hit this issue while using the disk cache. I noticed that the computed absolute path exceeds the limits of Windows 'lpMaximumComponentLength'[1]
This is a known python limitation[2]

The code below should help you recreate this issue on Windows. If you reduce the path length by a character, then this works.

k = 'MZQWY43FEBZWC3TDOR2WC4TJMVZTUIDTORXXE2LFOMQGM4TPNUQHI2DFEB2HE33VMJWGKZBAORSXE4TJORXXE2LFOMQG6ZRAONXXK5DIEBQXG2LBEBWWKZLOMFVXG2DJEBUXSZLSJFWWCZ3FONHWMZTFOJZUS5DFNVAXI5DSNFRHK5DFONIHE33NN52GS33OKN2W23LBO'

recfile.open(k, mode='wb', path='c:/temp/applications/sixapp/cache/abc/123456789012')
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-50-731a54413e60> in <module>()
----> 1 recfile.open(k, mode='wb', path='c:/temp/applications/sixapp/cache/abc/123456789012')

d:\code\git\web2py\gluon\recfile.pyc in open(filename, mode, path)
     63         if mode.startswith('w') and not os.path.exists(os.path.dirname(fullfilename)):
     64             os.makedirs(os.path.dirname(fullfilename))
---> 65     return file(fullfilename, mode)
     66
     67

IOError: [Errno 2] No such file or directory: 'c:/temp/applications/sixapp/cache/abc/123456789012\\135\\0d6\\MZQWY43FEBZWC3TDOR2WC4TJMVZTUIDTORXXE2LFOMQGM4TPNUQHI2DFEB2HE33VMJWGKZBAORSXE4TJORXXE2LFOMQG6ZRAONXXK5DIEBQXG2LBEBWWKZLOMFVXG2DJEBUXSZLSJFWWCZ3FONHWMZTFOJZUS5DFNVAXI5DSNFRHK5DFONIHE33NN52GS33OKN2W23LBO'

The resolution is to prepend '\\\\?\\' to the computed absolute path - in the case we are running this on Windows.


[1]: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx#maxpath
[2]: http://bugs.python.org/issue18199

[Niphlod]

or reduce the key to an hash.

[Kiran Subbaraman]

That is not something I control - reducing the key to a hash. But it looks like the recfile.py code does try to generate a hash for the incoming filename (key).
In any case, the issue remains on Windows. I'll see if I can suggest a patch.

[Niphlod]

it should . every file system has limits.

[Leonel Câmara]

The only fix I can think of is that recfile in windows does not allow keys that would lead to the idiotic windows path limit. We cannot use regular hash/truncate at the cache or another web2py level in this situation as collisions are not supposed to happen since that could lead to weird random bugs and we also need the hash to be reversible.

I suggest we don't fix this since recfile sort of already does this by throwing the exception, this is also clearly something that should either be fixed at the windows or python level, probably Python as it is supposed to abstract this kind of things and this is a serious leak in the abstraction in my opinion. Meanwhile this is a concern for windows users that they will have to address at the application level much as users of other OSes have to deal with other problems when the abstractions leak.

[Kiran Subbaraman]

I agree this is a limitation on the Python+Windows systems. I side-stepped the issue with what @niphlod mentioned "Just hash the key at the application level"
Maybe can be addressed via a 'Best practice' note in the documentation? - "If you are on Windows, then try to keep the key value for your cache small, by hashing the key value", or something to this effect.

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

On Wed, 04-11-2015 6:18 PM, Leonel Câmara wrote:

The only fix I can think of is that recfile in windows does not allow keys that would lead to the idiotic windows path limit. We cannot use regular hash/truncate at the cache or another web2py level in this situation as collisions are not supposed to happen since that could lead to weird random bugs and we also need the hash to be reversible.

I suggest we don't fix this since recfile sort of already does this by throwing the exception, this is also clearly something that should either be fixed at the windows or python level, probably Python as it is supposed to abstract this kind of things and this is a serious leak in the abstraction in my opinion. Meanwhile this is a concern for windows users that they will have to address at the application level much as users of other OSes have to deal with other problems when the abstractions leak.

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.