PUT and post data

[Thibault Jouannic]

Hi django devs,

I'm looking for some news about the ticket #12635.

https://code.djangoproject.com/ticket/12635

Today, it's still impossible to easily access post data when sending a
PUT request. Since I'm trying to build a restful web service using
django, this is quite a problem.

It's been told in another thread to use the `request.raw_post_data`
attribute, however I cannot think of a valid reason for why put data
could'nt be has easily accessible as post data.

I see in the ticket that the 1.4 milestone was deleted. Was the ticket
abandoned?

I've seen a lot of discussions and tickets about rest, and I know it's
a complex issue, but since I'm kinda new here, it's hard to know what
is the current state of the problem.

Regards,
Thibault J

[Ian Clelland]

"GET data" and "POST data" are just the terms that are used for data that appears in the URL query string, and the HTTP request body, respectively.  They don't actually have anything to do with the request method used*

If you receive a POST (or a PUT, or DELETE) at http://www.example.com/web_service?param=value, for example, then request.GET will be populated with {'param': 'value'}, no matter what the method is.

Similarly, if you attach a body to a PUT request, then it will be available just as in a POST.

request.POST is designed to work with HTML form submissions; data in either x/www-form-urlencoded or multipart/form-data, and it contains any key/value pairs parsed out of that sort of data. That's usually not what you attach to a PUT request, though (and occasionally, it's not what you attach to a POST, either). In that case, request.raw_post_data is the way to get at the unprocessed request body.

What you are after is just the body of the HTTP request, and it happens to be called 'raw_post_data' in django. It's a bit of an unfortunate name, since there is no difference between the payload of a POST request and that of a PUT request, but that's what it is.

Ian

 

* RFC-2616 prohibits a body on certain kinds of requests, so in practice, you won't see "POST data" on a GET or DELETE request, but you could see "GET data" on pretty much anything.

--
Regards,
Ian Clelland
<clel...@gmail.com>

[Carl Meyer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Thibault,

Thanks for checking in on this ticket. It's helpful to get a clear
resolution rather than leaving things hanging in an unclear state.

On 12/01/2011 11:14 AM, Thibault Jouannic wrote:
> I'm looking for some news about the ticket #12635.
>
> https://code.djangoproject.com/ticket/12635
>
> Today, it's still impossible to easily access post data when sending a
> PUT request. Since I'm trying to build a restful web service using
> django, this is quite a problem.
>
> It's been told in another thread to use the `request.raw_post_data`
> attribute, however I cannot think of a valid reason for why put data
> could'nt be has easily accessible as post data.

The valid reason is outlined by Malcolm Tredinnick in a thread linked
from that ticket [1], and by Ian Clelland just now. request.POST is a
special-case for a request body submitted with the
"x-www-form-urlencoded" or "multipart/form-data" content types. This is
common for POST because these content types are used by web browsers to
submit HTML forms.

However, web browsers do not submit PUT requests from HTML forms (and
they aren't likely to ever do so, as that proposed HTML5 feature was
removed [2]). Therefore, there's no reason to think form-urlencoded is
any more common than any other content type for PUT requests (in fact,
in my experience it's rare to find a web service that expects
form-urlencoded; JSON and XML are much more common). So there is no
reason to privilege form-urlencoded by adding a special case for it.

If you do want your web service to accept form-urlencoded or
multipart/form-data PUT requests, you can still use Django's QueryDict
and/or MultiPartParser to do the heavy lifting for you.

> I see in the ticket that the 1.4 milestone was deleted. Was the ticket
> abandoned?
>
> I've seen a lot of discussions and tickets about rest, and I know it's
> a complex issue, but since I'm kinda new here, it's hard to know what
> is the current state of the problem.

The ticket was in "Design Decision Needed" state, which isn't very clear
- - it means the core developers haven't yet made a decision on whether it
needs to be addressed or not.

I've now closed the ticket wontfix, based on the above reasoning.
Hopefully that helps clarify its status.

Carl

[1]
http://groups.google.com/group/django-developers/browse_thread/thread/771238a95ceb058e/
[2] http://www.w3.org/TR/2010/WD-html5-diff-20101019/#changes-2010-06-24
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7X5l0ACgkQ8W4rlRKtE2cAQACgmo9k/hCLtCK+fgvChJzqRSV7
9WQAoLox8Ok0EuqUSprxl0uG2e78zVpN
=gvqm
-----END PGP SIGNATURE-----

[Thibault Jouannic]

Thank you for taking the time to answer.

On 1 déc, 21:41, Carl Meyer <c...@oddbird.net> wrote:
> However, web browsers do not submit PUT requests from HTML forms (and
> they aren't likely to ever do so, as that proposed HTML5 feature was
> removed [2]).

Yeah, I was aware of that decision. The reason I had the problem is
because I was trying to build a ROA web service [1], and used some js
and middleware to make my browser perform some PUT and DELETE
requests.

Anyway, I agree with your decision, according to the W3C's design
decision.

Thank you for making things clearer.

[1] http://shop.oreilly.com/product/9780596529260.do

[Thibault Jouannic]

Actually, I've been thinking about this issue, read again the bug
report and current thread, and I'd like to re-open the discussion, if
you don't mind.

> request.POST is a
> special-case for a request body submitted with the
> "x-www-form-urlencoded" or "multipart/form-data" content types. This is
> common for POST because these content types are used by web browsers to
> submit HTML forms.

So, don't you think the `request.POST` variable should be processed
for every request with the "x-www-form-urlencoded" content type,
whatever the http verb is? Even if form urlencoded web services are
not common, it is still the easiest way to send a request to a web
server.

Here's a few examples.

1) Using curl http://pastie.org/2979317

2) Using httplib2 http://pastie.org/2979319

I understand that browsers should not send PUT requests, and I agree
that if the requests comes as json or xml, the parsing is up to the
developer. But in the case of an urlencoded request, why would django
make PUT requests' processing harder, forcing the user to parse the
querystring by himself (which is not really well documented, as far as
I know)?

If there is a valid explanation, I would be happy to hear it.
Otherwise, I could provide a patch to solve the issue. Let me know.

Regards,
Thibault J

[Florian Apolloner]

Hi,

On Wednesday, December 7, 2011 9:33:52 AM UTC+1, Thibault Jouannic wrote:

So, don't you think the `request.POST` variable should be processed
for every request with the "x-www-form-urlencoded" content type,
whatever the http verb is?

That would be really odd imo and confusing as hell if you find PUT/DELETE/HEAD/OPTIONS/whatever data in request.POST
 

But in the case of an urlencoded request, why would django

make PUT requests' processing harder, forcing the user to parse the
querystring by himself (which is not really well documented, as far as
I know)?

put_data = QueryDict(self.raw_post_data, encoding=your_encoding)
not really hard if you ask me.
 

If there is a valid explanation, I would be happy to hear it.
Otherwise, I could provide a patch to solve the issue. Let me know.

Stuffing it into POST is a no go, adding an extra PUT/OPTIONS/<whatever http verb allows entitity body> is something we don't like either. Given that the fix in Client code is a oneliner I think it's not really an issue.

Cheers,
Florian

[Thibault Jouannic]

Hi,

> That would be really odd imo and confusing as hell if you find
> PUT/DELETE/HEAD/OPTIONS/whatever data in request.POST

If I'm not wrong, only the PUT method can specify POST like
parameters.

> put_data = QueryDict(self.raw_post_data, encoding=your_encoding)
> not really hard if you ask me.

Well, fix is not that simple, since you have to handle the multipart/
form-data case, (which is the default encoding used in test client):

def
get_put_data(request):
content_type =
request.META.get('CONTENT_TYPE')
if
content_type.startswith('multipart'):
parser = MultiPartParser(request.META,
StringIO(request.raw_post_data), [], request.encoding) #nowhere in
docs.djangoproject.com
post, files =
parser.parse()

else:
post =
QueryDict(request.raw_post_data)
return post

I can work with that. My main concern is the lack of clarity in the
documentation. The testing doc asserts that « Client.put() [...] Acts
just like Client.post() except with the PUT request method. » When I
used Client.put in my tests, I was very surprised to find an empty
request.POST. Checked request.PUT ? Nope ! I had to browse a few
threads and bug reports to understand what was happening. Then, I had
to browse django's code to write the method above.

> Stuffing it into POST is a no go, adding an extra PUT/OPTIONS/<whatever
> http verb allows entitity body> is something we don't like either. Given
> that the fix in Client code is a oneliner I think it's not really an issue.

Then, maybe a variable named something like FORM_DATA would be less
error prone? Anyway, I think that discussion should be summarized
somewhere in the documentation, to prevent any further WTF?! effect.

Regards,
Thibault J.

[Florian Apolloner]

Hi,

On Wednesday, December 7, 2011 11:09:38 AM UTC+1, Thibault Jouannic wrote:

If I'm not wrong, only the PUT method can specify POST like
parameters.

Before writing my answer I did look on w3c and at least OPTIONS is allowed to have a body (can't tell you if it has to be a specific contenttype, but I doubt that).

Well, fix is not that simple, since you have to handle the multipart/
form-data case, (which is the default encoding used in test client):

Agreed, I was mostly referring to  your curl example which shouldn't issue multipart I guess ;)

I can work with that. My main concern is the lack of clarity in the
documentation.

+1 one, it would be good to know when what is filled (as people seem to appear to have problems with the current form).
 

 

Then, maybe a variable named something like FORM_DATA would be less
error prone?

Right, but that would make the distinction between GET and POST disappear, or at least I would expect GET params to be in it if I submitted a form etc…
Either way, since form-data is usually not the common way for stuff like this people might ask why there is no JSON_DATA etc…

Anyway, I think that discussion should be summarized
somewhere in the documentation, to prevent any further WTF?! effect.

Please add a patch to the tracker, since it always best if people with the problem write it as they know what's missing.

 

[Thibault Jouannic]

Hi again,

> Please add a patch to the tracker, since it always best if people with the
> problem write it as they know what's missing.

Done.

https://github.com/django/django/pull/87

Hope I did everything right.

I've read the submitting patches doc, but is a git pull request
accepted? Otherwise, Should I upload a patch in the closed ticket, or
open a new one?

Regards,
Thibault J

[Adrian Holovaty]

On Wed, Dec 7, 2011 at 8:25 AM, Thibault Jouannic <tee...@gmail.com> wrote:
> I've read the submitting patches doc, but is a git pull request
> accepted?

For now, no. We'll be moving to GitHub in the near future, but
currently we're using Subversion/Trac. Upload a patch via the ticket
tracker (code.djangoproject.com).

Adrian

[Carl Meyer]

Fwiw, for the last number of months we've been telling people that pull requests are fine in lieu of uploaded patch, but the pull request still needs to be linked from a Trac ticket (so Trac remains the single source of truth).

Carl
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Adrian Holovaty <adr...@holovaty.com> wrote:

-- 
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Carl Meyer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/07/2011 07:25 AM, Thibault Jouannic wrote:
> https://github.com/django/django/pull/87
>
> Hope I did everything right.
>
> I've read the submitting patches doc, but is a git pull request
> accepted? Otherwise, Should I upload a patch in the closed ticket, or
> open a new one?

Oh, just noticed this question. Generally it's best to err on the side
of opening a new ticket when a previous one has been closed, unless it's
exactly the same issue and the closing was simply in error. In this
case, it's not the same thing, as we're adding documentation of the
current behavior, not adding request.PUT as the previous ticket requested.

Thanks for the documentation patch!

Carl

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7f4KIACgkQ8W4rlRKtE2f6TACgqQGglLk4rpiMF+lkhnfQwKLs
dMYAnjuGZG8i/aLVltQxlcCtoJU+/Cu1
=QStu
-----END PGP SIGNATURE-----