streaming upload

[umrzyk]

hi there,
i would like to give my users ability to upload a huge (i.e. 10-20 MB)
files, mostly images. standard uploading using django newforms and
FileField is at the moment not an option. it kills my
server immediately. recently i read about streaming upload, and.. wow,
that's it! so i patched my django sources with 2070-r7728.patch (no
conflicts) and gave it a try. here are the problems:

- for small files (i didn't check what is a small file, i guess it's
something < 2.5MB as mentioned in doc) i got a validation message:
"the
submitted file is empty"

- for larger files (i tried 11MB) i got an error from server: "413
Request Entity Too Large"

i'm sure i'm doing something wrong, actually i'm the beginner. could
you
suggest where should i start searching? Or could you help me with a
sample snippet that would upload the file in a given directory?

running server is nginx 0.7.1 with mod_wsgi

thanks,
jm.

[Julien]

Hi,
There is a patch [1] which is almost ready and that should be merged
in Django very soon. You should give it a try and let the developers
know on the dev-list [2].

[1] http://code.djangoproject.com/ticket/2070
[2] http://groups.google.com/group/django-developers/browse_thread/thread/573a81affc5f09cc#

[Graham Dumpleton]

On Jun 30, 6:15 am, umrzyk <umr...@googlemail.com> wrote:

You may have to to ask the author of nginx mod_wsgi whether streaming
uploads is even possible.

I vaguely remember from discussions, that due to nginx mod_wsgi being
an event driven system, rather than using a threaded model, that it
may have to read complete request content into memory or onto disk
before it even executes the application. If it is into memory, then
what you are trying to do may not help as far as memory usage.

Graham

[umrzyk]

On 30 Cze, 03:51, Graham Dumpleton <Graham.Dumple...@gmail.com> wrote:
> You may have to to ask the author of nginx mod_wsgi whether streaming
> uploads is even possible.
>
> I vaguely remember from discussions, that due to nginx mod_wsgi being
> an event driven system, rather than using a threaded model, that it
> may have to read complete request content into memory or onto disk
> before it even executes the application. If it is into memory, then
> what you are trying to do may not help as far as memory usage.
>
> Graham

i have just consulted this issue with author of mod_wsgi and that
looks a little bit different. nginx does not cache whole the request
body in memory but instead (depending on size) makes use of temporary
file. please take a look at 'client_body_buffer_size' parameter if you
are interested in details.

at last i also managed to get rid of "413 Entity too large" error. it
was enough to increase 'client_max_body_size' parameter in nginx
config.

but i'm still unsure of one thing. while uploading small files i get
in request.FILES instance of InMemoryUploadedFile, for large files
this is TemporaryUploadedFile and that looks correct. Following is
what documentation says about uploaded files handling:

,----[http://code.djangoproject.com/git/?p=django;a=blob_plain;f=docs/
upload_handling.txt;hb=refs/heads/2070-streaming-uploads]
|
| Putting it all together, here's a common way you might handle an
uploaded file::
|
| def handle_uploaded_file(f):
| destination = open('some/file/name.txt', 'wb')
| for chunk in f.chunks():
| destination.write(chunk)
`----

and that is true for InMemoryUploadedFile as it has a chunks() method.
TemporaryUploadedFile does not have this method, so I cannot use above
snippet to handle both of them, or am i wrong? if so, what is the
'right' way to move uploaded file (no matter if it is InMemory- or
Temporary- uploaded) to defined directory?

thanks,
jm.

[Graham Dumpleton]

On Jul 1, 7:48 am, umrzyk <umr...@googlemail.com> wrote:
> On 30 Cze, 03:51, Graham Dumpleton <Graham.Dumple...@gmail.com> wrote:
>

> > You may have to to ask the author of nginxmod_wsgiwhether streaming

> > uploads is even possible.
>
> > I vaguely remember from discussions, that due to nginxmod_wsgibeing
> > an event driven system, rather than using a threaded model, that it
> > may have to read complete request content into memory or onto disk
> > before it even executes the application. If it is into memory, then
> > what you are trying to do may not help as far as memory usage.
>
> > Graham
>

> i have just consulted this issue with author ofmod_wsgiand that

You can only use methods defined in WSGI specification as being valid
for wsgi.input. Thus, you need to ignore the actual type of object and
should not base decisions on it. See:

http://www.python.org/dev/peps/pep-0333/#input-and-error-streams

Anything outside of read(), readline(), readlines() and __iter__() is
non standard and not portable. WSGI read() also has to be supplied a
length when calling it.

Graham

Graham

[umrzyk]

guys,
i have just set up apache (with mod_wsgi) to verify this issue and it
occured that django complains about empty submitted file independently
from running server. so I guess somethig is wrong with either my code
(most probably...) or with the patch.
Once again, the problem is that i cannot upload small files (request
body hold in memory). form validator returns immediately "the
submitted file is empty" message in this case. large files upload
without any problems.
my form code:

class ArticleForm(forms.Form):
title = forms.CharField(max_length=100)
content = forms.FileField()

def save(self,commit = True,user=None,f=None):
article = Article(created=user, content=f)
if commit:
article.save()
return article

view code:

def workspace(request):
if request.method == 'POST':
form = ArticleForm(request.POST, request.FILES)
if form.is_valid():
article =
form.save(user=request.user,f=request.FILES['content'])
else:
form = ArticleForm()
return render_to_response('workspace.html', {'user': request.user,
'form': form})

and models code:

class Article(models.Model):
title = models.CharField(max_length=100)
content = models.FileField(upload_to="articles")


any suggestions what is wrong here?

thanks,
jm.

[Mike Axiak]

Hi jm,

I've been trying to reproduce your problem with no success. Can you
please post your nginx configuration (a safe version, obviously)?

Also, if you stop by #django, my username is axiak. It might be easier
to track this down in IRC.

Cheers,
Mike

On Jul 2, 6:26 pm, umrzyk <umr...@googlemail.com> wrote:
> On 29 Cze, 22:15, umrzyk <umr...@googlemail.com> wrote:
>
>
>
> > hi there,

> > i would like to give my users ability touploada huge (i.e. 10-20 MB)

> > files, mostly images. standard uploading using django newforms and
> > FileField is at the moment not an option. it kills my

> > server immediately. recently i read about streamingupload, and.. wow,

> > that's it! so i patched my django sources with 2070-r7728.patch (no
> > conflicts) and gave it a try. here are the problems:
>
> > - for small files (i didn't check what is a small file, i guess it's
> > something < 2.5MB as mentioned in doc) i got a validation message:
> > "the
> > submitted file is empty"
>
> > - for larger files (i tried 11MB) i got an error from server: "413
> >   Request Entity Too Large"
>
> > i'm sure i'm doing something wrong, actually i'm the beginner. could
> > you
> > suggest where should i start searching? Or could you help me with a

> > sample snippet that woulduploadthe file in a given directory?

>
> > running server is nginx 0.7.1 with mod_wsgi
>
> > thanks,
> > jm.
>
> guys,
> i have just set up apache (with mod_wsgi) to verify this issue and it
> occured that django complains about empty submitted file independently
> from running server. so I guess somethig is wrong with either my code
> (most probably...) or with the patch.

> Once again, the problem is that i cannotuploadsmall files (request

[umrzyk]

On 29 Cze, 22:15, umrzyk <umr...@googlemail.com> wrote:

> hi there,
> i would like to give my users ability to upload a huge (i.e. 10-20 MB)
> files, mostly images. standard uploading using django newforms and
> FileField is at the moment not an option. it kills my
> server immediately. recently i read about streaming upload, and.. wow,
> that's it! so i patched my django sources with 2070-r7728.patch (no
> conflicts) and gave it a try.

[cut]
ok, the issue is resolved now. after an hour of IRC chatting with
Mike, it occured that was problem neither with apache/nginx nor with
my code. I have just pulled older revision from svn and applied 2070
patch by myself. Although there were no conflicts then, something was
not working. Today i updated django from svn once again. as Mike said
2070 has been commited to the trunk few days ago so there is no need
to apply it again by hand. Now everything works like a charm.

thanks for your help,
michal.