Field('upload') : problem with file name

[Loïc]

Dear All,

I have a problem with Field('upload). It seems that some file names produce an error in DAL.

Let's take a "test" app to reproduce it.

1)  A simple model :

db = DAL('mysql://user:pass@myServer')
db.define_table('file',
   Field('title', label=T('Title'), notnull=True),
   Field('file', 'upload', uploadfolder=os.path.join(
        request.folder,'static','uploaded_files'
        ), notnull=True, autodelete=True, label=T('File')),
   format='%(title)s'
   )

2) a controller :

def files():
    a_file = db.file(request.args(0))
    crud.settings.update_deletable=False
    if len(request.args) and a_file:
        crud.settings.update_deletable = False
        form = crud.update(db.file,a_file,next='files_list')
    else:
        form = crud.create(db.file)
    return dict(a_file=a_file, form=form)

3) A view "files.html":

{{extend 'layout.html'}}
{{ if a_file: }}
    <h1>{{=T('Edit file')}}</h1>
{{ else: }}
    <h1>{{=T('Add a file')}}</h1>
{{ pass }}
{{=form}}

Then go on http://127.0.0.1:8000/test/default/files/
Try to upload a file : everything works fine

Create a simple file (regardless the type, it can be a txt file or anything else) but named  Audi A6 Avant 3.0 V6 TDI 240 DPF Quattro Tiptronic S Line (Break) 10_2009, Packs et options - Cote automobile - autoplus.fr.txt or take the attached file

Try to upload this file : you got the fllowing error

Error ticket for "test"

Ticket ID

127.0.0.1.2013-03-16.11-43-07.109e09df-378c-4e89-96b1-175c300f70d6

<type 'exceptions.IOError'> [Errno 2] No such file or directory: 'D:\\Google Drive\\loic\\web2py\\web2py\\applications\\test\\static\\uploaded_files\\file.file.accd0d98e01c0b64.41756469204136204176616e7420332e30205636205444492032343020445046205175617474726f2054697074726f6e69632053204c696e652028427265616b292031305f323030392c205061636b73206574206f7074696f6e73202d20436f7465206175746f6d6f62696c65202d20.txt'

Version

web2py™	(2, 4, 2, 'stable', datetime.datetime(2013, 3, 4, 3, 26, 21))
Python	Python 2.7.2: C:\Python27\python.exe (prefix: C:\Python27)

Traceback

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.

Traceback (most recent call last): File "D:\Google Drive\loic\web2py\web2py\gluon\restricted.py", line 212, in restricted exec ccode in environment File "D:/Google Drive/loic/web2py/web2py/applications/test/controllers/default.py", line 87, in <module> File "D:\Google Drive\loic\web2py\web2py\gluon\globals.py", line 193, in <lambda> self._caller = lambda f: f() File "D:/Google Drive/loic/web2py/web2py/applications/test/controllers/default.py", line 84, in files form = crud.create(db.file) File "D:\Google Drive\loic\web2py\web2py\gluon\tools.py", line 3631, in create **attributes File "D:\Google Drive\loic\web2py\web2py\gluon\tools.py", line 3572, in update detect_record_change=self.settings.detect_record_change): File "D:\Google Drive\loic\web2py\web2py\gluon\sqlhtml.py", line 1459, in accepts field.uploadfolder) File "D:\Google Drive\loic\web2py\web2py\gluon\dal.py", line 9228, in store dest_file = open(pathfilename, 'wb') IOError: [Errno 2] No such file or directory: 'D:\\Google Drive\\loic\\web2py\\web2py\\applications\\test\\static\\uploaded_files\\file.file.accd0d98e01c0b64.41756469204136204176616e7420332e30205636205444492032343020445046205175617474726f2054697074726f6e69632053204c696e652028427265616b292031305f323030392c205061636b73206574206f7074696f6e73202d20436f7465206175746f6d6f62696c65202d20.txt'

I think the file name contains not allowed characters but I don't understand what is the exact problem...
Can anyone explain me?

I'm on Windows7 pro x64, Python 2.7.2, Web2py 2.4.2, MySQL 5.5.29

Thank you

[wwwgong]

You forgot to create the uploadfolder

[Loïc]

No, I have created the uploadforder.
I can upload every files I want, excepted a file which is named Audi A6 Avant 3.0 V6 TDI 240 DPF Quattro Tiptronic S Line (Break) 10_2009, Packs et options - Cote automobile - autoplus.fr.txt regardless its content

[LightDot]

If you're sure it's the file name, make some test cases, see what fails exactly and what doesn't... Usual suspects from the file name in question would be dots, parentheses, dashes, commas, spaces... I don't see any others. Or perhaps a hidden rogue character, like a space that isn't a space...?

Regards,
Ales

[Loïc]

After some tries, I discovered that the problem is not in the characters.

The problem is the file name length
If  filename length <= 76 characters it works
If filename length > 76 characters the error ticket is raised...

[Niphlod]

if **feels** strange cause file is stored with a computed hash .... did you spot the issue on the code while figuring out the name on the disk or in the part where the record gets stored in the table ?

[Loïc]

I think I have found the problem. It's not a web2py issue but a python issue on windows :

in dal.py", line 9228 there is the following line :

dest_file = open(pathfilename, 'wb')

If the filename is too big, the hash pathfilename length is about 275 characters
I found a thread on stackoverflow which mentionned that In the Windows API the maximum path length is limited to 260 characters.
http://stackoverflow.com/questions/4677234/python-ioerror-exception-when-creating-a-long-file

So maybe I should reduce the filename length before upload
When I use an 'upload' field, is it possible to automatically rename the file before upload?

[Niphlod]

that seems a bug. Unless you're forcing it, web2py should save a file in

folder/tablename_field.somehash.originalextension

i.e. whatever length the filename originally is, it gets stored with a somewhat "constant" filename length.

Can you pack a minimal app to reproduce the issue ?

[Niphlod]

I was wrong. the first part of the "hash" is a uuid, the second is a base16 encoding of the original filename.

Seems that you need some logic to cut down long files if you plan to host on Windows.
http://web2py.com/books/default/chapter/29/07?search=uploads#Storing-the-original-filename

[Loïc]

Thank you Niphlod, I'll try this...

[Scott Hunter]

This is not only an issue with Windows; any filesystem with a filename limit (however big) can hit this problem.  As this is all being handled by web2py, and the user is selecting the file to upload, where would this "logic" go, and why isn't web2py handling it?

- Scott

[Massimo Di Pierro]

This is handled by web2py. The filenames are truncated according to the size of the uploadfield. This defaults to 512. In new versions of windows the make file length has been reduced so you have to do

Field(name, 'upload', size=200)

and all filenames will be limited to 200 bytes.

[Scott Hunter]

Where is this spelled out in the documentation?  I could not find it.

- Scott

[Massimo Di Pierro]

Nowhere. We only state that Field(..., lenght=....) is a parameter. We do not say that for "upload" fields this is also the length of the tmp filename.

[Scott Hunter]

Is this an oversight or a conscious decision?  If I understand you, if one's filesystem allowed filenames greater than 512 characters, long-enough but still legal names will still be truncated, and thus thwart the ability to extract the original file name.  That could be a nasty surprise.

P.S. As I understand it, it is a permanent filename.

[Massimo Di Pierro]

web2py renames the uploaded filenames as tablename+code+hash.extension

The total len(tablename+code+hash+extension)<=512

This means len(hash) <= 512 - len(tablename+code+extension)

The hash is a base64 encoded of the original filename.

If this hash exceeds 512 - len(tablename+code+extension) the file is truncated.

Mind this is still a lot of characters.

Massimo

[Scott Hunter]

I'm sorry, I wasn't clear.  I was asking about the lack of documentation: was there a decision not to document this (if so, I'd be curious as to its basis), or was it just an oversight that will hopefully be remedied in the future?

- Scott

[Richard Vézina]

Scott, you can improve the documentation if you feel it need to : https://github.com/mdipierro/web2py-book

I think we can't be  responsible for all the issue related to filesystem differencies... I think Massimo is really kind to pin point the issue like that... And since we have the information we should make it available...

:)

Richard

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Wabbajack]

Hi Massimo,

Still has this issue... :(

I am using 2.14.6 web2py running in windows7 with pydal 17.03 and python 2.7


I have this filename 54 characters (including '.pdf ')




This is the field on my db.py


After i add a record including an upload file  this is the Error

Adding record including upload file




Need help on resolving this issue since i cannot control the filename length of the user will upload
Also i check the length newfilename length and including its path hope it helps




The error is directing to this file



I also tried changing my field with length value but still error upon adding files with upload


Thank you in advance,
Jak

[Dave S]

On Wednesday, May 24, 2017 at 9:37:17 PM UTC-7, Wabbajack wrote:

Hi Massimo,

Still has this issue... :(

I am using 2.14.6 web2py running in windows7 with pydal 17.03 and python 2.7


I have this filename 54 characters (including '.pdf ')




This is the field on my db.py


After i add a record including an upload file  this is the Error

Adding record including upload file




Need help on resolving this issue since i cannot control the filename length of the user will upload
Also i check the length newfilename length and including its path hope it helps




The error is directing to this file



I also tried changing my field with length value but still error upon adding files with upload


Thank you in advance,
Jak

 

If you're opening a file on the server, then the length of the absolute path may matter.  If your application folder is deeply nested, than that could be a problem.  On Windows, your choices

• move your web2py (and application) folder nearer the root of the filesystem
• assign a drive letter to the path (shortening the total path) ... this may be the old alias command, but it's been a while since I did this
• use Window's version of links to provide a shorter path (I think this became "real" in XP, but again I haven't done this in a while).

Note that you do have some control over the length of the filename:

• you can reject names that are too long, usually done in the form
• you can rename the file to a shorter name, and separately track the original name

Note that my experience includes having the web2py folder in a path like

c:/users/mememe/web2py_win/web2py-2.14.6/web2py

That's a fair-sized headstart on total pathlength,

but so far I haven't had any users (any of me) produce a filename that caused a problem.

On a linux system where actual users upload files, the longest name (as stored) is 235 characters, and the path is 57 characters.  The filename is actually generated by a non-web program that we control, and sent via libcurl.  The name encodes some information relevant to the client.  The 235 length is after web2py hashes the name used on the client.

/dps

 

[Wabba Jak]

Hi Dave,

Yes it helps... :)

Putting the web2py directory near the Drive path.. (T_T)

Also i will try your suggestion editing the filename before the upload process in the Form

Thanks it greatly helps..

/dps

 

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---

You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/ee6NBnj_TyU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.

[Andrew Rogers]

>>> Field(name, 'upload', size=200)

I think 'size' should be 'length'. The latter worked for me and the former didn't.