receiving PNG file in django

[Larry Martell]

I have a falcon server that I am trying to port to django. One of the
falcon endpoints processes a request that contains a PNG file sent
with content_type = 'application/octet-stream'. It writes the data to
a file maintaining the correct PNG structure.

The falcon code does this:

form = cgi.FieldStorage(fp=req.stream, environ=req.env)

and then writes the png like this:

fd.write(form[key].file.read())

I cannot figure out how to do the same thing in django. When my view
is called the data in request.POST[key] has already been decoded to
unicode text and it's no longer valid png data.

How can I do this with django? Should/can I use cgi.FieldStorage? The
request I get (of type django.core.handlers.wsgi.WSGIRequest) does not
have a stream method. I'm sure there's some way to do this, but I have
not come up with anything googling.

[Aron Griffis]

Hi Larry,

I think you'll find what you're looking for in request.FILES, see https://docs.djangoproject.com/en/1.10/ref/request-response/#django.http.HttpRequest.FILES and https://docs.djangoproject.com/en/1.10/ref/files/uploads/#uploaded-files

This is the same as using cgi.FieldStorage to parse files from a submitted multipart/form-data. You can then call .read() on the UploadedFile object, or better yet use .chunks() to copy to your final destination without overwhelming memory.

-Aron

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CACwCsY5m4fdczXdQ0gYX61gBe0BYQ-ZoZ1R_vpkrGEzv%2B2-vBg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Larry Martell]

When I get the request, request.FILES is empty. Yet the content type
is multipart/form-data and the method is POST:

(Pdb) print request.META['CONTENT_TYPE']
multipart/form-data;
boundary="boundary_.oOo._NzEwNjIzMTM4MTI4NjUxOTM5OQ==MTY2NjE4MDk5Nw=="

(Pdb) print request.META['REQUEST_METHOD']
POST

(Pdb) print request.FILES
<MultiValueDict: {}>

The individual files are being sent with content-type
'application/octet-stream', and it looks like I get the contents of
the files converted to unicode:

(Pdb) type(request.POST['right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'])
<type 'unicode'>

I've tried to decoding it with:

unicodedata.normalize('NFKD', request.POST[key]).encode('ascii','ignore'))

But that did not create a valid PNG file. It seems that something
mistakenly encoded to unicode. What would be doing that?

Any suggestions on how I can make this work?

[Larry Martell]

On Fri, Aug 19, 2016 at 4:58 PM, Bill Freeman <ke1...@gmail.com> wrote:
> If the original data truly isn't on the request object somewhere, you might
> be able to grab it earlier by writing a request midleware. You might have
> to put it pretty early in the chain.
>
> The first thing to try is changing the content_type to something more
> appropriate,

What would be more appropriate?

> and see if the data shows up in request.FILES (but my
> experience here is older, and things may or may not still work this way in
> newer django's.
>
> Somewhat harder is that request.POST[key]'s conversion to unicode might be
> reversible, though not necessarily with an existing codec.
>
> Finally, you might suck down and decode the request data yourself in the
> middleware, and either save the file and return a response yourself, or fix
> the request up so that the rest of the middlewares and the view won't be
> upset.
>
> Good luck, and I hope someone gives you a better answer.
>
> Bill

>
> On Fri, Aug 19, 2016 at 4:17 PM, Larry Martell <larry....@gmail.com>
> wrote:
>>

>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to django-users...@googlegroups.com.

[Michal Petrucha]

On Mon, Aug 22, 2016 at 10:11:49AM -0400, Larry Martell wrote:
> When I get the request, request.FILES is empty. Yet the content type
> is multipart/form-data and the method is POST:
>
> (Pdb) print request.META['CONTENT_TYPE']
> multipart/form-data;
> boundary="boundary_.oOo._NzEwNjIzMTM4MTI4NjUxOTM5OQ==MTY2NjE4MDk5Nw=="
>
> (Pdb) print request.META['REQUEST_METHOD']
> POST
>
> (Pdb) print request.FILES
> <MultiValueDict: {}>
>
> The individual files are being sent with content-type
> 'application/octet-stream', and it looks like I get the contents of
> the files converted to unicode:
>
> (Pdb) type(request.POST['right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'])
> <type 'unicode'>

Are you certain the request contains correct headers for files? In
particular, look at the “Content-Disposition” header; for each file,
it should contain the “filename” attribute; otherwise, Django will
treat it as a regular (non-file) form field [1]. As an example, the
header could look like this::

Content-Disposition: form-data; name="file"; filename="filename.png"

> I've tried to decoding it with:
>
> unicodedata.normalize('NFKD', request.POST[key]).encode('ascii','ignore'))
>
> But that did not create a valid PNG file. It seems that something
> mistakenly encoded to unicode. What would be doing that?

By this point, it's too late – Django has already treated the field as
a string field, so it's already called
``force_text(data, encoding, errors='replace')`` [2], which means the
value you have there has already lost some information that is
impossible to get back [3].

And even if it was reversible, performing a Unicode normalization, and
then throwing away everything non-ASCII would most certainly not be
the correct way, since it would mangle about one half of the input
data into ASCII.

> Any suggestions on how I can make this work?

You'll have to make sure that the request is correct, I'm afraid.

Good luck,

Michal


[1]: https://github.com/django/django/blob/a8f957797d8035a542cdb20b03aaebd81b9529e2/django/http/multipartparser.py#L638-L641
[2]: https://github.com/django/django/blob/a8f957797d8035a542cdb20b03aaebd81b9529e2/django/http/multipartparser.py#L207
[3]: https://docs.python.org/2/library/codecs.html#codec-base-classes

[Larry Martell]

On Mon, Aug 22, 2016 at 11:00 AM, Michal Petrucha
<michal....@koniiiik.org> wrote:
> On Mon, Aug 22, 2016 at 10:11:49AM -0400, Larry Martell wrote:
>> When I get the request, request.FILES is empty. Yet the content type
>> is multipart/form-data and the method is POST:
>>
>> (Pdb) print request.META['CONTENT_TYPE']
>> multipart/form-data;
>> boundary="boundary_.oOo._NzEwNjIzMTM4MTI4NjUxOTM5OQ==MTY2NjE4MDk5Nw=="
>>
>> (Pdb) print request.META['REQUEST_METHOD']
>> POST
>>
>> (Pdb) print request.FILES
>> <MultiValueDict: {}>
>>
>> The individual files are being sent with content-type
>> 'application/octet-stream', and it looks like I get the contents of
>> the files converted to unicode:
>>
>> (Pdb) type(request.POST['right-carotidartery:63B2E474-D690-445F-B92A-31EBADDC9D93.png'])
>> <type 'unicode'>
>
> Are you certain the request contains correct headers for files? In
> particular, look at the “Content-Disposition” header; for each file,
> it should contain the “filename” attribute; otherwise, Django will
> treat it as a regular (non-file) form field [1]. As an example, the
> header could look like this::
>
> Content-Disposition: form-data; name="file"; filename="filename.png"

How would I access the Content-Disposition for each file? In the
headers for the request that field does not exist.

[Michal Petrucha]

What I'm suggesting is that you look at the raw HTTP request that's
going over the wire, or investigate the client that's making the
request. I'm not sure if there's any reasonable way to work around
this on the Django side.

[Larry Martell]

On Mon, Aug 22, 2016 at 11:14 AM, Michal Petrucha

Thanks so much. I solved this by changing the client to set the file
and filename fields each part of the multipart and then I was able it
iterate through request.FILES and successfully write the files as PNG.

[Larry Martell]

On Mon, Aug 22, 2016 at 12:11 PM, Bill Freeman <ke1...@gmail.com> wrote:
> The most appropriate thing is for the source to use the correct content
> type, in which case I believe that Django just works (it certainly used to).
> But I'm assuming that you can change that. The stuff I suggested were hacks
> and as such are things you do in a darkened room and keep secret. Note that
> you want to recognize when it's one of these uploads before applying the
> chosen hack(s).

One man's hack is another man's elegant solution ;-)