Support POST of application/json content type

[Stefan Berder]

Hi,
I looked around the list and couldn't find any mention of this subject.

In `django.http.request.HttpRequest._load_post_and_files()` there is explicit mention of two content type ('multipart/form-data' and 'application/x-www-form-urlencoded'), any other content type will get empty values for self._post.

Given that a lot of user form interaction is now happening through 'XMLHttpRequest', I think that the 'application/json' content type should be supported. A lot of javascript libraries will use json as the default format:
* angularjs: http://docs.angularjs.org/api/ng.$http, see "Setting HTTP Headers"
* emberjs: https://github.com/emberjs/data/blob/master/packages/ember-data/lib/adapters/rest_adapter.js#L974
* backbone: http://backbonejs.org/#Sync
* jquery: http://api.jquery.com/jQuery.ajax/ (the only one using 'application/x-www-form-urlencoded' by default)

I'm trying primarily to create a discussion on the subject and am ready to provide the code for it as I had to write it. This would help avoid hacks to handle the request object in my view.

I know there are some apps to handle API construction (django-tastypie, django-rest, django-piston and certainly others) they use either request wrappers or request handling in their views.

Stefan

[Marc Tamlyn]

+1 to providing a better way of accessing JSON encoded post data. -1 to it being munged into request.POST - I feel this is unintuitive.

A nice possibility could be a simple piece of middleware which detects `application/json` and attaches request.json as an attribute, if it can decode it. This attribute could actually be made lazy so as not to introduce any real overhead on other pages, or on views already decoding the data directly from request.data.

There is a (somewhat stalled) ticket for a JsonResponse class at #17942 (https://code.djangoproject.com/ticket/17942).

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

[Curtis Maloney]

What do people think of an extensible registration approach to handling request content types?

Something along the lines of ('POST', [list of content types], class to handle it)

This would mean adding a request.JSON would be simple for the projects that want it, and even overriding how the existing content types are handled would be possible. [I haven't seen a use case for such yet, but it's a wild world out there :)]

--

Curtis

[Curtis Maloney]

Someone remind me never to post to list before I've had my second coffee? :)

Of course the above would trivially be handled using middleware.

--

C

[Tom Christie]

Hi Stefan,

Sure, I'd be interested in seeing us improve how we deal with JSON requests and responses.

My preference would be to introduce a request parsing and response rendering API that allows us to support not just JSON, but any media type that has a parser installed for it.  (I've commented on some of this before, here, although I think I'm warming towards the idea that it's probably about time we started addressing at least some of this in core.)

Unsurprisingly I'd suggest the same general approach that is used in REST framework - A lazy `request.DATA` attribute (or similar) that when accessed, inspects the media type on the request, and parses the request stream with an appropriate parser if possible.  The installed parsers can be configured globally, or on a per-request basis.  The existing multipart and form-encoded parsing behaviour would no longer be a special case baked directly into the request object, but instead be the default installed parsers.

Taking this approach makes it trivial to write views that can handle both JSON and form data, and providing a proper parser API makes it easy for developers to package up and share their own parser implementation, such as YAML, XML and MsgPack.  (And, looking forwards, JSON-based media types such as hypermedia types.)

In REST framework this behaviour is (by necessity) implemented in a Request object that wraps the underlying HttpRequest, but the same basic implementation can be applied to implementing it directly in the Request object, and would be somewhat easier.

I'm interested to see Marc suggesting middleware specifically for handling JSON requests.  That'd work, and would be a simple approach.  My reservations with that would be:

* We'd not be making it any easier for users to deal with request parsing generally.

* There's no way to write views that deal with request data in an agonistic way, and dealing with differing media types would require switching based on the media type in the view itself.  For example, the generic views would still only support form data.  As another example, if you wanted to add, say, MsgPack support to your application, you'd need to re-write all your views.

From my point of view this is already a solved problem, and I'd really like to see a generic approach to handling request data, and a corresponding approach to rendering responses into an appropriate media type.

 All the best,

  Tom

[Stefan Berder]

Tom,
I agree that the middleware solution is not the best and in my mind creates fragmentation of the same type of request handling.

A generic content-type framework would make the request parsing more flexible, stronger to change in the future and open the door to any type.

I'm curious about the choice of request.DATA though, when doing a POST request, I expect the data to be in the POST dictionary and nowhere else. Creating a request.DATA attribute would break compatibility with old code and seem to me less intuitive. Where should request.FILE go in that case? Would request.POST just be a call to request.DATA?

Stefan

[Tom Christie]

> Creating a request.DATA attribute would break compatibility with old code and seem to me less intuitive.

The implementation would ensure `request.POST` would still be populated for form data.

There wouldn't have to be any backwards incompatible changes, and usage of `request.DATA` (or whatever better name there might be, perhaps simply `request.data`) would be entirely optional for using with generic request parsing, instead of form data only.

> Where should request.FILE go in that case

request.FILES would be populated by form data as normal, and would be empty on JSON or other submissions.  In order to support multipart parsing the request parsing API would need to provide for parsers that support file upload, in a similar way that REST framework currently does.

> Would request.POST just be a call to request.DATA?

That's an open question, but I'd probably expect it to only return data if the request contains multipart or url-encoded form data.  (Ie. the behaviour wouldn't change.)

Cheers,

  Tom

[Marc Tamlyn]

The thing with request.POST is that it's kinda unintuitive when extended to other HTTP methods (e.g. PUT). This is why the old request.raw_post_data was renamed request.body.

request.POST would behave in the expected traditional web way of picking up form encoded POST data, which would also be available in request.DATA as well, but request.DATA is the "new" way of doing it. Personally, I'd lowercase it though, to remove confusion with the PHP $POST $GET $REQUEST which we mimic on the request object. The generally have different use cases anyway - one for complex web things and the other for standard web browsers.

(the above is what tom said...)

Tom - what else do you have in DRF's Request that you would need?

--

[Jonathan Slenders]

Would that mean that the object returned by request.DATA/POST/whatever could be a different type, depending on what the user posted?

I don't want to see code like:

if isinstance(request.DATA, YamlObject): ...
elif isinstance(request.DATA, dict): ...

although, I'm not sure how any view could handle any random content-type...

[Curtis Maloney]

To weight in from an author of a different API tool....

In django-nap there's a simple "get_request_data" method which, essentially, determines how to parse the request according to the content type.  However, currently each API only supports a single serialiser format [and HTTP Form encoding] so there's little guessing involved.

However, I wouldn't want to see the same on Request, unless there was also a direct way to imply the type you want.  For instance, if I get a request that's in XML, and I ask for request.JSON, I'd like it to either yield empty or raise an error.  Whereas when I didn't case, accessing request.DATA would make a best guess.

So I see two paths... either you use a "Decode it for me according to content-type" interface, or a "Treat is as this type, and fail predictably if it's not" one.

The current GET/POST is, of course, the latter.  And there's no reason we can't have both :)

--

Curtis

[Tom Christie]

In REST framework, we expose the accepted renderer as an `request.accepted_renderer` property.

This allows you to switch based on the media type when needed by either doing something like:

    if request.accepted_renderer.format == 'yaml':

        # do something specific

or like this:

    if request.accepted_renderer.media_type == 'application/json':

        # do something specific

That allows you to do media type specific stuff if needed, but generally you'll just access `request.DATA`, and not need to be concerned about exactly what media type was decoded for you.  (Of course the API specifics could be open to bike shedding.)  In response to Jonathan's particular query, REST framework supports a YAML parser, and there's never any need to do specific branching to detect it, because it just returns native python datatypes: lists, dicts, etc... exactly the same as the JSON parser does.

And yes, different media types can and do have different capabilities (eg. form data decodes into QueryDict objects, rather than regular dicts) but you're able to switch on that if specifically needed.

I'm not at all keen on the idea of injecting differing attribute names on the request based on the media type that is decoded, partly because it just feels like an odd API to me, but more concretely because it unnecessarily ties your view logic to your parser classes, and doesn't allow you to easily support views that accept multiple content types.  (The combination of either form data or json being the most common.) 

> what else do you have in DRF's Request that you would need

The bits of API we'd need to support something like this:

* `request.DATA` / `request.data` for accessing the parsed data.

* `request.parsers` or similar for modifying the parsers prior to parsing on a per-view basis.  (Note the parallel to request.upload_handlers)

* `request.accepted_parser` or similar for branching on media type if needed.

* A setting that controls the default set of installed parsers.

There are a couple of other bits of behaviour/API, but they're not related to request parsing.  I wouldn't mind discussing those as well at some point, but I don't want to go off on a tangent just yet.

The other core thing would be the question of if dealing with request parsing also means we should necessarily also tackle response rendering (and ideally, content negotiation) at the same time.  I don't really mind whether that'd be seen as a requirement or not, but it is important that we have a consistent design between both parsing and rendering.

Cheers all,

  Tom

[S Berder]

Gents,
to sum it up, arguments made and details of how I see the
implementation of a response/request encode/decode framework:

* need a pluggable interface so current content-types are supported
(`application/x-www-form-urlencoded`, `multipart/form-data`), new
types (`application/json`), custom and future types
(`application/vnd.foobar+json` anybody? See
http://developer.github.com/v3/media/#api-v3-media-type-and-the-future
for example, `application/msgpack`, `application/protobuf`,
`application/capnproto`, etc).
* decoder/encoder map (content-type, decoder) should be smart to
handle patterns like `text/*` or `application/*xml*` and match things
like `Accept: application/json, text/plain, * / *`
* choice of decoder would be made on the Content-Type header, maybe
supporting a raw by default so data is just passed in case of unknown
content type.
* decoder/encoder should be available through `request` and `response` objects.
* decoded data structure (python object) would be stored in `request.data`
* first step is to support requests, next step is to handle responses
with the same pluggable functionality and coherent API.
* A sensible default for response Content-type would be `text/html;
charset=UTF-8`. It should be made available through a setting entry
anyway

Some questions though:

* why keep data and files separated, I see no good reason for this
except mimicking PHP's structure. An uploaded file comes from a named
input, I hope to find it in request.data (why do a common structure
otherwise). I might be missing something but nothing indicates a real
need for this in django/http/request.py
* isn't more or less any data sent to your backend representable as a
dict or object with dict access modes? I try to think about
occurrences where some data would not have a 'name'.

I'm sure I left some obvious things out like more settings and others
but this will become obvious while coding.

If no strong objection, I'm ready to start in that direction and
provide code sometime next week.

Stefan

> You received this message because you are subscribed to a topic in the

> Google Groups "Django developers" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/s8OZ9yNh-8c/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.



--

http://www.bonz.org/
/(bb|[^b]{2})/

[Curtis Maloney]

On 9 September 2013 19:50, S Berder <sbe...@gmail.com> wrote:

Gents,
to sum it up, arguments made and details of how I see the
implementation of a response/request encode/decode framework:

* need a pluggable interface so current content-types are supported
(`application/x-www-form-urlencoded`, `multipart/form-data`), new
types (`application/json`), custom and future types
(`application/vnd.foobar+json` anybody? See
http://developer.github.com/v3/media/#api-v3-media-type-and-the-future
for example, `application/msgpack`, `application/protobuf`,
`application/capnproto`, etc).
* decoder/encoder map (content-type, decoder) should be smart to
handle patterns like `text/*` or `application/*xml*` and match things
like `Accept: application/json, text/plain, * / *`
* choice of decoder would be made on the Content-Type header, maybe
supporting a raw by default so data is just passed in case of unknown
content type.
* decoder/encoder should be available through `request` and `response` objects.
* decoded data structure (python object) would be stored in `request.data`
* first step is to support requests, next step is to handle responses
with the same pluggable functionality and coherent API.
* A sensible default for response Content-type would be `text/html;
charset=UTF-8`. It should be made available through a setting entry
anyway

You should also have access to the decision made by the data parser as to which parser was used, instead of having to infer it yourself from the content type header.

 

Some questions though:

* why keep data and files separated, I see no good reason for this
except mimicking PHP's structure. An uploaded file comes from a named
input, I hope to find it in request.data (why do a common structure
otherwise). I might be missing something but nothing indicates a real
need for this in django/http/request.py

True, there's some added complexity [small as it is] in forms because File fields need to look elsewhere for their values.

 

* isn't more or less any data sent to your backend representable as a
dict or object with dict access modes? I try to think about
occurrences where some data would not have a 'name'.

I frequently send JSON lists of data to my APIs...

 

--

Curtis

[S Berder]

Indeed, that's the 4th point of my list, maybe it's not clear as it is
but this would be supported.

>>
>> Some questions though:
>>
>> * why keep data and files separated, I see no good reason for this
>> except mimicking PHP's structure. An uploaded file comes from a named
>> input, I hope to find it in request.data (why do a common structure
>> otherwise). I might be missing something but nothing indicates a real
>> need for this in django/http/request.py
>
>
> True, there's some added complexity [small as it is] in forms because File
> fields need to look elsewhere for their values.
>
>>
>> * isn't more or less any data sent to your backend representable as a
>> dict or object with dict access modes? I try to think about
>> occurrences where some data would not have a 'name'.
>>
>
> I frequently send JSON lists of data to my APIs...

Ok, was a bit short sighted on this one, still thinking in terms of
form bound data, it was a long day here in Shanghai. I suppose that
the kind of python object you receive is not so important as you
should do data validation anyway. Your earlier concern about checking
for different content-types doesn't apply to the solution I have in
mind as to whatever data representation you have at the beginning, you
should get a very similar object after decoding. What I mean is if you
send the *same* data through Yaml or JSON, the object in request.data
should be the same or extremely close. I say extremely close because
I'm thinking about xml that is always way more verbose than the others
and *might* add more data to the resulting object. (hint: I don't like
XML, don't need it in what I do and last used it ~8/9 years ago in a
disastrous explosion of SOAP and unix/microsoft interfaces)

Stefan

[S Berder]

In the absence of strong objection, I will start working on this base.

[Tom Christie]

> why keep data and files separated

Mostly because that's the way it already works, so...

* request.data would essentially provide a strict superset of the functionality that request.POST provides.  In general you'd be able to replace `request.POST` with `request.data` anywhere and seemlessly start supporting JSON or other data without any other changes required.

* Form expect the data and files to be provided separately which would be awkward otherwise.

> In the absence of strong objection, I will start working on this base. 

Sure thing.  As it happens, I was also considering taking a crack at this in the coming weeks, so please do follow up on this thread linking to your repo if you start working on it, so myself and others can track any progress.  (And perhaps collaborate.)

Cheers :)

  Tom

[S Berder]

Tom,
I get the context and that form is tied to that behavior but outside of "that's how it is now" is there any technical reason for this? I can't read forms code at the moment but will do tonight and will look at how FILES is used in there. I'm not usually afraid by impacts of it's the "right thing to do". I think it would make for a stronger interface of you could simply find your data in request.data.

I will create a branch in my fork on github and will send it here for progress.

Stefan, from my mobile
--
http://www.bonz.org/
/(bb|[^b]{2}/

--

[abh2134]

Hi all,

I would love to be involved with this feature. 

My suggestion is to do the following:

• Check requst.is_ajax()
• Check request.META.get('CONTENT_TYPE').count('application/json')
• Parse request.body using django.utils.simplejson.loads
• ... and set request.JSON to the result 

I have a small piece of middleware that implements this procedure and have posted it as a gist, https://gist.github.com/abhillman/6910689.

If there is interest in using my code, I would be very happy to write some unit tests.

abh

[Tom Christie]

I've created the ticket 21442 to track 'Configurable request parsing'.

I've also made a first pass at refactoring the current request parsing into a generic interface that can support arbitrary parsers,

and pushed the current status to the ticket_21442 branch on my GitHub account.

It's likely that the API will need a few design decisions along the way, so I'd suggest anyone who's interested in contributing to this work makes sure to follow the ongoing work in that branch.

Cheers all,

  Tom

[Dmitry Mugtasimov]

It looks like there is a lot of work to do for full featured content-type support. Meanwhile if you need just to integrate Django + Django REST Framework and django-oauth-plus for PUT-parameters support, please, take a look at this thread https://groups.google.com/forum/#!topic/django-rest-framework/679fOg0QpzI and this post https://groups.google.com/d/msg/django-rest-framework/679fOg0QpzI/2qeQGdzq-FsJ in particular.

понедельник, 18 ноября 2013 г., 17:42:31 UTC+4 пользователь Tom Christie написал: