SQLFORM upload type and long file+path names

[David Braun]

I unexpectedly ran into this problem, and it took me a while to figure
out what was going on. I'm wondering if anyone has any suggestions on
how to handle it.

I'm using the 'upload' field type with SQLFORMs to upload files to the
server. However, for certain files with longer names, I get this
exception:

<type 'exceptions.IOError'>([Errno 2] No such file or directory: 'C:\
\Users\\---------------\\Documents\\------\\---------------\
\--------------------------\\web2py\\applications\\webcollection\
\databases\\..\\uploads\\usermodel_file_store.file.9dbf56ec79482a5b.
68616c6f6c7a2d646f742d636f6d2d6e36342d736e65732d796f756d696768746265747279696e67746f686172642e6a7067.jpg')

(Dashes are hiding my personal folder names, they don't actually
appear like that)

I couldn't for the life of me figure out what was going on, until I
luckily tried uploading a file with a short name, which worked
perfectly.

Doing some digging, it looks like Windows has a maximum length of 260
characters for a file's combined path and name (http://
windows.microsoft.com/en-US/windows-vista/File-names-and-file-name-
extensions-frequently-asked-questions). Doing some experimenting in
windows explorer, this appears accurate.

So, between my long base path and the long encoded file names, I'm
hitting that limit.

All that out of the way, I guess I have two questions:
1) Is there an easy way to "overload" the default behavior of the
SQLFORM so it stores the filename in the database, instead of encoding
it in the renamed-file? I see the Field class has a custom_store/
retrieve variable, could I just duplicate the default DAL store/
retrieve functions and add my extra file name logic? Or is there a
better way?
2) Is there any way for web2py to better identify this exception? I
could imagine some people scratching their heads for a quite a while
on it.

[Massimo Di Pierro]

> 1) Is there an easy way to "overload" the default behavior of the
> SQLFORM so it stores the filename in the database, instead of encoding
> it in the renamed-file? I see the Field class has a custom_store/
> retrieve variable, could I just duplicate the default DAL store/
> retrieve functions and add my extra file name logic? Or is there a
> better way?

One thing you can do it is

Field('name','upload',length=260)

it will truncate the generated filename to 260. That means it will not
be able to reconstruct the full original name but only part of it.
Anyway, give it a try and let us know.

> 2) Is there any way for web2py to better identify this exception? I
> could imagine some people scratching their heads for a quite a while
> on it.

Python raises an IOError. This could have many possible causes. I
changed the code in trunk to re-raise the exception with more info.
Check if this is acceptable.

Massimo

[Anthony]

On Thursday, November 24, 2011 12:06:24 AM UTC-5, Massimo Di Pierro wrote:

> 1) Is there an easy way to "overload" the default behavior of the
> SQLFORM so it stores the filename in the database, instead of encoding
> it in the renamed-file? I see the Field class has a custom_store/
> retrieve variable, could I just duplicate the default DAL store/
> retrieve functions and add my extra file name logic? Or is there a
> better way?

One thing you can do it is

Field('name','upload',length=260)

it will truncate the generated filename to 260.

Note, I think using the 'length' arg in this way requires trunk. In the current stable version, it automatically limits the filename to 200 characters (plus extension). If you set length to less than 200, I'm not sure what happens.

In trunk, length defaults to 512, but if you set it lower, it will limit the entire filename (including extension) to that length. Note, the filename is of the form

[table].[field].[16-char uuid fragment].[b16encoded original filename].[extension]

If that ends up longer than 'length', it gets truncated from the end (though the extension is not truncated). So, depending on the table and field names, you need about 40 characters even without encoding the original filename at all.

Also, note that the 'length' argument applies to the filename only -- it does not include the full path length, so you have to account for that separately when setting the length (maybe we should have it apply to the full path length when the file is being stored on the filesystem).

Anthony