Advancing the "content negotiation" and "modernising request object" proposals.

[Carlton Gibson]

Hi all.

I'm looking for a high-level sanity check if you would.

I've been trying to see a way forward through a nest of issues around two concrete proposals:

1. Adding "content negotiation" to the request object, allowing automatical parsing of different content types, such as JSON, as well as allowing that to work for all request methods, rather than just POST.

2. Modernising the API for the request object, adding attributes such as `request.data`, and `request.query_params`, etc., rather than the uppercase POST, GET, and so on.

The first is a major stepping stone towards having (JSON or other) API support in core — the "merge DRF into core" request that comes up frequently. (The other main side of that would be a review of serialization and forms, in light of developments such as Pydantic, attrs/cattrs, and django-readers, but that is **not** on topic here.) This was first suggested in 2011, but has made little progress in that time. [0][1]

[0]: https://groups.google.com/g/django-developers/c/4c4xT3ULNLk
[1]: https://code.djangoproject.com/ticket/21442

The second Adam Johnson proposed 2020, and was nearly merged bar Mariusz
**blinking** at the size of the distruption, particularly for documentation
throughout the community, for no change in behaviour. [2][3]

There was an inconclusive discussion about whether we right there[4] but, at the time I linked the modernisation to the content negotiation issue, as the feature needed to pay for the change.

[2]: https://groups.google.com/g/django-developers/c/Kx8BfU-z4_E/m/lFXTF0IMCQAJ
[3]: https://code.djangoproject.com/ticket/32259
[4]: https://forum.djangoproject.com/t/technical-board-vote-on-ticket-32259-modernize-request-attribute-names/10255

Digging further into the history, with a mind to move these issues forward, having **not** merged Adam's patch first time gives us the needed pathway forward, I hope.

I think there have been two reasons the content negotiation suggestion has not progressed:

1. It's been all or nothing. Numerous times it's been requested to **just** add JSON handling, but that's been bounced back to the full proposal, adding customisable parsers and so on, which has then stalled.[5][6]

[5]: https://code.djangoproject.com/ticket/27415
[6]: https://code.djangoproject.com/ticket/32646

2. There's a backwards compatibility concern, particularly with multipart request bodies, where currently you'd get a string, which you'd then try to parse yourself, not expecting an already parsed dictionary, for example.[7]

[7]: https://code.djangoproject.com/ticket/28678

The way around the backwards compatibility concern is to introduce a new code
pathway at `request.data` that handles things in the new way, whilst
deprecating `request.POST`, which could either be removed or become an alias to
`request.data` at the end of the deprecation period. (Given the behaviour
change, and django-upgrade ongoing development, I'd lean now toward removal, I
think.)

In order to get this done, I'd like to introduce this **without also solving the pluggable parsers issue** in the first version.

That is, I would like to add `request.data` to provide parsed data from the request body, for all request methods, together with `application/json` content type handling (and multipart parsing for `application/json` parts as well) **but** I would like to leave the configurable parsers step for a later iteration.

I think this would give most of the benefit, and allow us to (finally) make forward steps here. My hope it that this is addressable before the 4.2 feature freeze in January, but if not, OK, it hits 5.0 — at least it's in.

Folks needing other content types can parse request.body as they'd need to do now.
Having a list of request.parsers, configurable in e.g. View.setup(), or a middleware, or even a custom request class — essentially at any point before accessing request.data — would be the follow-up. Clearly, this would be good to have, but I feel like we've blocked on it so long, finding a way forward that allows it to be deferred would be sensible. (Prefetch evolved...)

Matching ``request.data``, if Adam will pick it up, the modernised request API would be
delightful. (The schedule leaves room for the code changes to come before the
documentation updates if needed.)


Does this seem like a reasonable plan? Thanks


Kind Regards,

Carlton

[charettes]

Hello Carlton,

This is not an area of the code base I'm heavily involved with but the increment approach you are proposing over this lack of feature support for basic content negotiation seems like a sane approach to gradually make the landscape better in this area without trying to get everything just right in a single stab.

Adding ``request.data`` with support limited to JSON bodies at first seems the minimal step to lay some foundations towards revisiting the inclusion of very core/HTTP centric features that are sadly only available in DRF at the moment.

+1 from me.

Cheers,

Simon

[Adam Johnson]

This first-step solution is good with me. It will allow everyone to switch to request.data (etc.). And there’d be a clear way to use your own logic to set request.data if needed: write a middleware (or view decorator, view class, etc.).

What should request.data be/do in the case of an unsupported content type? Currently request.POST returns an empty QueryDict. But DRF raises UnsupportedMediaType if it has no matching parser, which is translated into a 415 Unsupported Media Type response.

DRF’s behaviour feels more correct to me, since it allows terser views that don’t check the content type explicitly. But it’s less backwards compatible. I’m not sure which I prefer.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/daff8eee-3d35-45e4-83fe-98d6676b813an%40googlegroups.com.

[charettes]

> DRF’s behaviour feels more correct to me, since it allows terser views that don’t check the content type explicitly. But it’s less backwards compatible. I’m not sure which I prefer.

Given the .data attribute would be a new feature of the request object I assume we don't have any backward compatiblity concerns to worry about as long as we document the behaviour of .data properly and leave .POST unchanged? I would have a slight preference for raising an UnsupportedMediaType as well and letting that percolate to a 415 as it seems more correct from a content negotiation perspective.

[Adam Johnson]

I would have a slight preference for raising an UnsupportedMediaType as well and letting that percolate to a 415 as it seems more correct from a content negotiation perspective.

Thinking about it again I think I have a slight preference too.

I guess this would warrant adding a urlconf option for a custom handler415 view, like handler400 etc. ?

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/294ee7a7-de94-4e47-a7ec-0ac3510ce52dn%40googlegroups.com.

[st...@jigsawtech.co.uk]

I also agree with raising a UnsupportedMediaType and having custom handlers for 400, 415 is always useful IMO