file upload with flash 8 uploader

[spako]

hi

i'm building a site with which will let the public upload files (after
registration) to the site. i'm going to use flash to upload the files
since the files will be large and with the new flash you can see the
progress of an upload.

the uploader is in a view i have built. i'm expecting the upload to
appear in request.FILES but that does not happen and an error occurs,
is there something that needs to be set to allow this to happen? extra
cookie info that needs to be sent maybe?

i've tested the uploader with a php script so there's not a problem
with the mechinism.

[patrickk]

I´m having a multiple upload and I´m using:
for file in request.FILES.getlist('imageupload_file'):
...

[Malcolm Tredinnick]

On Fri, 2006-07-07 at 09:50 +0000, spako wrote:
> hi
>
> i'm building a site with which will let the public upload files (after
> registration) to the site. i'm going to use flash to upload the files
> since the files will be large and with the new flash you can see the
> progress of an upload.
>
> the uploader is in a view i have built. i'm expecting the upload to
> appear in request.FILES but that does not happen and an error occurs,
> is there something that needs to be set to allow this to happen? extra
> cookie info that needs to be sent maybe?

How does the Flash uploader (never used it) present the data to the
server side? Normally, a form with file uploads attached will be sent
with a MIME type of multipart/form-data and Django knows how to break up
such a message and parse the form data and the file data into separate
pieces. I could imagine that the Flash uploader might not construct it's
request that way, but I have no idea.

If you want to debug this from your code, have a look at
request.raw_data, to see what Django received, and request.META will
contain all of the HTTP headers in upper-cased format.

Regards,
Malcolm

[spako]

have gotten the following reqeust header, from a header monitor. i also
get an error prointed which i will paste below. also i've pasted the
view here, it doesn't seem like the request gets to the view. i have
tried hitting the view directly in the browser and that works fine.

--- view ----

def do_upload_video(request):
print request.raw_data
print request.META
return render_to_response('user/upload3.html')

---- request header -----

POST /upload/2/do/ HTTP/1.1
Accept: text/*
Content-Type: multipart/form-data;
boundary=----------cH2ei4cH2cH2cH2ei4KM7gL6Ef1Ij5
User-Agent: Shockwave Flash
Host: localhost:8000
Content-Length: 99446
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: sessionid=cda865cb732e46761b79a78f7ad0a87c

------------cH2ei4cH2cH2cH2ei4KM7gL6Ef1Ij5
Content-Disposition: form-data; name="Filename"

thumbs.JPG
------------cH2ei4cH2cH2cH2ei4KM7gL6Ef1Ij5
Content-Disposition: form-data; name="Filedata"; filename="thumbs.JPG"
Content-Type: application/octet-stream

ÿØÿà
.....
.....
.....
(¢€?ÿÙ
------------cH2ei4cH2cH2cH2ei4KM7gL6Ef1Ij5
Content-Disposition: form-data; name="Upload"
Submit Query
------------cH2ei4cH2cH2cH2ei4KM7gL6Ef1Ij5

---- error printed in console when request is run ----

Traceback (most recent call last):
File "c:\python24\lib\site-packages\django\core\servers\basehttp.py",
line 272, in run
self.result = application(self.environ, self.start_response)
File "c:\python24\lib\site-packages\django\core\servers\basehttp.py",
line 615, in __call__
return self.application(environ, start_response)
File "c:\python24\lib\site-packages\django\core\handlers\wsgi.py",
line 146, in __call__
response = self.get_response(request.path, request)
File "c:\python24\lib\site-packages\django\core\handlers\base.py",
line 102, in get_response
return self.get_technical_error_response(request)
File "c:\python24\lib\site-packages\django\core\handlers\base.py",
line 135, in get_technical_error_response
return debug.technical_500_response(request, *sys.exc_info())
File "c:\python24\lib\site-packages\django\views\debug.py", line 133,
in technical_500_response
return HttpResponseServerError(t.render(c), mimetype='text/html')
File "C:\Python24\lib\site-packages\django\template\__init__.py",
line 155, in render
return self.nodelist.render(context)
File "C:\Python24\lib\site-packages\django\template\__init__.py",
line 688, in render
bits.append(self.render_node(node, context))
File "C:\Python24\lib\site-packages\django\template\__init__.py",
line 716, in render_node
raise wrapped
TemplateSyntaxError: Caught an exception while rendering.

Original Traceback (most recent call last):
File "C:\Python24\lib\site-packages\django\template\__init__.py",
line 706, in render_node
result = node.render(context)
File "C:\Python24\lib\site-packages\django\template\defaulttags.py",
line 189, in render
value = bool_expr.resolve(context, True)
File "C:\Python24\lib\site-packages\django\template\__init__.py",
line 548, in resolve
obj = resolve_variable(self.var, context)
File "C:\Python24\lib\site-packages\django\template\__init__.py",
line 657, in resolve_variable
raise VariableDoesNotExist, "Failed lookup for key [%s] in %r" %
(bits[0], current) # missing attribute
File "c:\python24\lib\site-packages\django\core\handlers\wsgi.py",
line 62, in __repr__
return '<DjangoRequest\nGET:%s,\nPOST:%s,\nCOOKIES:%s,\nMETA:%s>' %
\
File "c:\python24\lib\site-packages\django\core\handlers\wsgi.py",
line 97, in _get_post
self._load_post_and_files()
File "c:\python24\lib\site-packages\django\core\handlers\wsgi.py",
line 75, in _load_post_and_files
self._post, self._files = http.parse_file_upload(header_dict,
self.raw_post_data)
File "c:\python24\lib\site-packages\django\http\__init__.py", line
52, in parse_file_upload
name_dict = parse_header(submessage['Content-Disposition'])[1]
File "c:\python24\lib\cgi.py", line 331, in parse_header
plist = map(lambda x: x.strip(), line.split(';'))
AttributeError: 'NoneType' object has no attribute 'split'

[07/Jul/2006 12:12:41] "POST /upload/2/do/ HTTP/1.1" 500 2880

[spako]

i've added some print statements in the django code like this:

Index: __init__.py
===================================================================
--- __init__.py (revision 3287)
+++ __init__.py (working copy)
@@ -39,6 +39,8 @@
return ''

def parse_file_upload(header_dict, post_data):
+ print post_data
+ print header_dict
"Returns a tuple of (POST MultiValueDict, FILES MultiValueDict)"
import email, email.Message
from cgi import parse_header

when i do an upload from the flash player now, the script seems to get
stuck in a loop, it keeps printing post_data and header_dict
forever.... i'm new to django and python so any help or pointing in the
direction to fix this will be appreciated!

here's one copy of what was printed in the console, minus the image
data:

------------gL6ei4cH2GI3cH2ae0gL6gL6ae0KM7
Content-Disposition: form-data; name="Filename"

tmp.jpg
------------gL6ei4cH2GI3cH2ae0gL6gL6ae0KM7
Content-Disposition: form-data; name="Filedata"; filename="tmp.jpg"
Content-Type: application/octet-stream

◄↓►♫►
....
....
....
┘
------------gL6ei4cH2GI3cH2ae0gL6gL6ae0KM7

Content-Disposition: form-data; name="Upload"
Submit Query

------------gL6ei4cH2GI3cH2ae0gL6gL6ae0KM7
{'HTTP_ACCEPT': 'text/*', 'HTTP_USER_AGENT': 'Shockwave Flash',
'HTTP_CONNECTION': 'Keep-Alive', 'HTTP_COOKIE':
'sessionid=cda865cb732e46761b79a78f7ad0a87c',
'HTTP_HOST': 'localhost:8000', 'HTTP_CACHE_CONTROL': 'no-cache',
'Content-Type': 'multipart/form-data;
boundary=----------gL6ei4cH2GI3cH2ae0gL6gL6ae0KM7'}

[Malcolm Tredinnick]

On Fri, 2006-07-07 at 11:42 +0000, spako wrote:
> i've added some print statements in the django code like this:
>
> Index: __init__.py
> ===================================================================
> --- __init__.py (revision 3287)
> +++ __init__.py (working copy)
> @@ -39,6 +39,8 @@
> return ''
>
> def parse_file_upload(header_dict, post_data):
> + print post_data
> + print header_dict
> "Returns a tuple of (POST MultiValueDict, FILES MultiValueDict)"
> import email, email.Message
> from cgi import parse_header
>
>
> when i do an upload from the flash player now, the script seems to get
> stuck in a loop, it keeps printing post_data and header_dict
> forever.... i'm new to django and python so any help or pointing in the
> direction to fix this will be appreciated!
>
> here's one copy of what was printed in the console, minus the image
> data:

[...]

This debugging output helped work out what is happening here. Thanks.

I'm not sure where the bug lies, but when the email module parses the
closing boundary marker in your form submission, it is adding a part to
the resulting object that does not have a Content-Disposition set (when
I try this at the command line, it looks like an empty email message
body).

We could work around this in Django by changing line 59 in
http/__init__.py to read:

if not submessage['Content-Disposition']:
continue

name_dict = parse_header(submessage['Content-Disposition'])[1]

# etc...

However, this still will not make your data available via request.FILES.
Later on in that function you see the comment:

# We assume all uploaded files have a 'filename' set.

and since you "Upload" block does not have a filename, it won't be
considered. There are ways we could rewrite that (e.g. using
submessage.get_param()) to work around this, though.

Might be worth filing a ticket to request this enhancement. I'm not 100%
sure of my conclusions here, so hopefully some people with more "in the
trenches" experiences with multi-part uploads will confirm or correct. I
tend to debug multipart problems as they arise, without ever really
having gotten a full grasp on all the RFCs involved; they make my head
hurt too much.

In the interim (after putting in the workaround above), you could parse
request.raw_data yourself via email.message_from_string(...) and get
access to your uploaded data that way if you don't want to modify your
local source.

Regards,
Malcolm

[arthur debert]

On macrodobe's site, there's a reference implementation of php
receiveing the flash upload. it's php, but you can have an idea of
where things are on the request:

http://livedocs.macromedia.com/flash/8/main/wwhelp/wwhimpl/common/html/wwhelp.htm?context=LiveDocs_Parts&file=00001591.html

The flash upload capabilitie is pretty nifty... I wonder if there's any
interest of creating a FileField(flash_uplload=True) option and have it
and django as an alternative.

spako:
currently uploads are kept in memory, not streamed. that means if you
are expecting large uploads (+2MB) and are on a shared server, you
might have issues (since most of them hava a memory limit your
processes can use).

patch 2070 is suppouse to work around this, but I couldn't get it
working here.

cheers
arthur

[spako]

thanks for the response guys. i fixed the problem to with something
similar to what you suggest Malcom. i started a new thread here:
http://groups.google.com/group/django-users/browse_thread/thread/baa6c6d64f59639e

that fixed the problem, seemed to will be sure to keep it together next
time! also i was able to use that file request.FILES. if picked up the
name in this line:

Content-Disposition: form-data; name="Filedata"; filename="thumbs.JPG"

so i access the file as request.FILES['Filedata']

arthur, thanks for the link. the files will be pretty big, think we
going to have a limit of 50mb. but we are running on a dedicated server
so might be able to get away with it. although i've also spotted that
path before and am going to try implement that, or something similar.