(Very) Large File Upload - mod_python MemoryError

[carole...@gmail.com]

I'm using a non-Django form for a very large file upload ( I've never
used FileField, because they used to give me problems, and this code
has worked forever for smaller files ).

It seems to work great with files 32MB and smaller... but anything
over that seems to throw this mod_python MemoryError. I am all for
using FTP, but the powers that be ( my boss/project manager ) would
like people to be able to upload a 150MB file via form post ( I know
it takes over an hour..nuts ).

This is my code:
fout = file
(os.path.join(getUploadDir(), filename), 'wb',10000)
while 1:
chunk =
fileitem.file.read(10000)
if not chunk: break
fout.write (chunk)
fout.close()


Anyone been successful w/ Django + mod_python and really large file
uploads? I saw something about a streaming file upload patch, but
I'm assuming I'd have to switch to FileField for that, and wanted to
see if anyone had tried it first...

Thanks for any info.
Carole

[Karen Tracey]

The code in Django that causes out-of-memory errors for large file uploads is the code that populates request.FILES -- this is code that runs before your view is even called so it doesn't matter if you are using a newforms.FileField or rolling your own form to handle uploaded files.  As it stands today, before ever calling your view Django will try to read all of the uploaded file data into memory, and that is where you run into out-of-memory problems.

So, I believe you will need the patch for large streaming uploads (http://code.djangoproject.com/ticket/2070) if you want to be able to handle uploading very large files.  This is a big patch and I'm not sure how close it is to getting into the code base, but I gave it (the last of the many patches listed in the ticket) a quick try and it worked fine for my app.  Given where this code hooks in I don't know that you would need to switch to using FileField, but I can't guarantee your current code will work since in my case I am using a newforms.FileField to upload files (though I am not using models.FileField).  Personally, though, I'd rather get whatever problems I had with FileField fixed than work around them, but since you didn't give any details on what the problems you ran into were it's hard to provide any guidance on how to do that.

Karen

[Mike Axiak]

I'm am one of the authors of the 2070 patch [1]. Indeed, it is what
will help you here.

However, I'm not sure how it will work with your code as I don't
really see the context and I'm not omniscient. However, here's how
you'd write to files in #2070::

from django.core.files.filemove import file_move_safe
file_obj = request.FILES['field_name']
if hasattr(file, 'temporary_file_path'):
file_obj.close()
file_move_safe(file_obj.temporary_file_path(),
destination_path)
else:
# We use file locks because with such large files collisions
*do* happen.
from django.core.files import filelocks
fp = open(destination_path, 'wb')
filelocks.lock(fp, filelocks.LOCK_EX)
for chunk in file_obj.chunk():
fp.write(chunk)
filelocks.unlock(fp)
fp.close()

Note that most of this code is in _save_FIELD_file in
django.db.models.base after patching #2070.
Also note that if you use the standard save_field_file interface, this
is already done for you. So you can write::

instance = SomeModel(...)
instance.save_field_file(request.FILES['field_name'].file_name,
request.FILES['field_name'], save=False)
instance.save()

And it will do what I outlined above.

I hope this helps! Let me know if you have any problems or further
questions. (Specifically, posting in #2070 will let me keep track of
it better.)

Cheers,
Mike

1: http://code.djangoproject.com/ticket/2070