I'm working on a REST-full API that allows an uploaded file to be PUT to a URL. I also need to support the X-Http-Method-Override header to turn a POST request into a PUT for clients (Flash, mobile etc) that can't do PUT.
At the moment I've got the API working with POST requests, and am now moving over to PUT + implementing the override.
I'm running into several places where it seems Django processes things differently if they come from a POST or a PUT. The most obvious is the HttpRequest (and derived classes) handling of _load_post_and_files(). This parses a multipart upload into a set of files in request.FILES and the rest of the POST data into request.POST, whenever request.POST is accessed.
However a similar transformation isn't applied to PUT requests - so accessing request.FILES finds nothing there, and there's no equivalent request.PUT.
This seems very brittle if anything goes near request.POST before the rest interface runs, since the request.POST access will parse the uploaded data which is then not available for reparsing - error messages such as "Cannot read more than the available bytes from the HTTP incoming data" then pop up.
It seems to me that Django should process POST and PUT requests the same - i.e. the request.FILES attribute should be initialised for either if a multipart content-type is detected.
Have I fundamentally misunderstood how this stuff should work?
On Sat, Jan 16, 2010 at 6:32 AM, Malcolm Box <malcolm....@gmail.com> wrote: > Hi,
> I'm working on a REST-full API that allows an uploaded file to be PUT to a > URL. I also need to support the X-Http-Method-Override header to turn a > POST request into a PUT for clients (Flash, mobile etc) that can't do PUT.
> At the moment I've got the API working with POST requests, and am now moving > over to PUT + implementing the override.
> I'm running into several places where it seems Django processes things > differently if they come from a POST or a PUT. The most obvious is the > HttpRequest (and derived classes) handling of _load_post_and_files(). This > parses a multipart upload into a set of files in request.FILES and the rest > of the POST data into request.POST, whenever request.POST is accessed.
> However a similar transformation isn't applied to PUT requests - so > accessing request.FILES finds nothing there, and there's no equivalent > request.PUT.
> This seems very brittle if anything goes near request.POST before the rest > interface runs, since the request.POST access will parse the uploaded data > which is then not available for reparsing - error messages such as "Cannot > read more than the available bytes from the HTTP incoming data" then pop up.
> It seems to me that Django should process POST and PUT requests the same - > i.e. the request.FILES attribute should be initialised for either if a > multipart content-type is detected.
> Have I fundamentally misunderstood how this stuff should work?
On first inspection, I don't think you've got anything fundamentally wrong. Support for PUT and the other 'exotic' HTTP request methods is one area where Django support is a little bit patchy.
This is mostly an artefact of the vast majority of browsers providing exactly zero support for requests other that GET and POST. As a result Django is (by necessity) really good at handling GET and POST; PUT, DELETE et al are all essential parts of the full HTTP spec, but support in Django isn't quite as comprehensive.
This is one of those areas where patches are definitely welcome.
> wrote: > On Sat, Jan 16, 2010 at 6:32 AM, Malcolm Box <malcolm....@gmail.com> > wrote: > <snip> > > It seems to me that Django should process POST and PUT requests the same > - > > i.e. the request.FILES attribute should be initialised for either if a > > multipart content-type is detected.
> > Have I fundamentally misunderstood how this stuff should work?
> On first inspection, I don't think you've got anything fundamentally > wrong. Support for PUT and the other 'exotic' HTTP request methods is > one area where Django support is a little bit patchy.
> Well that's good to know! > This is mostly an artefact of the vast majority of browsers providing > exactly zero support for requests other that GET and POST. As a result > Django is (by necessity) really good at handling GET and POST; PUT, > DELETE et al are all essential parts of the full HTTP spec, but > support in Django isn't quite as comprehensive.
> This is one of those areas where patches are definitely welcome.
> I'm happy to provide some patches - presumably a bug report + patch is the
way to go.
Which raises the question of what the correct behaviour should be, so the bug can report that it's not happening and the patch can implement.
As a first pass:
"On a PUT request, Django should populate request.PUT and request.FILES in the same way as for POST requests. FILES will be initialised if the content is multipart, and the PUT dictionary will have key/value pairs if the uploaded content included url-encoded form entries.
If the format of the datastream is not understood, it will be left in _raw_post_data as previously, and req.FILES and req.PUT initialised to empty hashes."
Does this sound reasonable as a bug/feature description?
On Mon, Jan 18, 2010 at 8:14 AM, Malcolm Box <malcolm....@gmail.com> wrote: > On Sat, Jan 16, 2010 at 4:58 AM, Russell Keith-Magee > <freakboy3...@gmail.com> wrote:
>> On Sat, Jan 16, 2010 at 6:32 AM, Malcolm Box <malcolm....@gmail.com> >> wrote: >> <snip> >> > It seems to me that Django should process POST and PUT requests the same >> > - >> > i.e. the request.FILES attribute should be initialised for either if a >> > multipart content-type is detected.
>> > Have I fundamentally misunderstood how this stuff should work?
>> On first inspection, I don't think you've got anything fundamentally >> wrong. Support for PUT and the other 'exotic' HTTP request methods is >> one area where Django support is a little bit patchy.
> Well that's good to know!
>> This is mostly an artefact of the vast majority of browsers providing >> exactly zero support for requests other that GET and POST. As a result >> Django is (by necessity) really good at handling GET and POST; PUT, >> DELETE et al are all essential parts of the full HTTP spec, but >> support in Django isn't quite as comprehensive.
>> This is one of those areas where patches are definitely welcome.
> I'm happy to provide some patches - presumably a bug report + patch is the > way to go.
As always.
> Which raises the question of what the correct behaviour should be, so the > bug can report that it's not happening and the patch can implement.
> As a first pass:
> "On a PUT request, Django should populate request.PUT and request.FILES in > the same way as for POST requests. FILES will be initialised if the content > is multipart, and the PUT dictionary will have key/value pairs if the > uploaded content included url-encoded form entries.
> If the format of the datastream is not understood, it will be left in > _raw_post_data as previously, and req.FILES and req.PUT initialised to empty > hashes."
> Does this sound reasonable as a bug/feature description?
Sounds reasonable to me. I can't think of any obvious reason that the handling for PUT should differ from POST.
> wrote: > On Mon, Jan 18, 2010 at 8:14 AM, Malcolm Box <malcolm....@gmail.com> > wrote: > > On Sat, Jan 16, 2010 at 4:58 AM, Russell Keith-Magee> Which raises the > question of what the correct behaviour should be, so the > > bug can report that it's not happening and the patch can implement.
> > As a first pass:
> > "On a PUT request, Django should populate request.PUT and request.FILES > in > > the same way as for POST requests. FILES will be initialised if the > content > > is multipart, and the PUT dictionary will have key/value pairs if the > > uploaded content included url-encoded form entries.
> > If the format of the datastream is not understood, it will be left in > > _raw_post_data as previously, and req.FILES and req.PUT initialised to > empty > > hashes."
> > Does this sound reasonable as a bug/feature description?
> Sounds reasonable to me. I can't think of any obvious reason that the > handling for PUT should differ from POST.
> On Mon, Jan 18, 2010 at 8:14 AM, Malcolm Box <malcolm....@gmail.com> wrote: >> On Sat, Jan 16, 2010 at 4:58 AM, Russell Keith-Magee >> <freakboy3...@gmail.com> wrote:
>>> On Sat, Jan 16, 2010 at 6:32 AM, Malcolm Box <malcolm....@gmail.com> >>> wrote: >>> <snip> >>>> It seems to me that Django should process POST and PUT requests the same >>>> - >>>> i.e. the request.FILES attribute should be initialised for either if a >>>> multipart content-type is detected.
>>>> Have I fundamentally misunderstood how this stuff should work?
>>> On first inspection, I don't think you've got anything fundamentally >>> wrong. Support for PUT and the other 'exotic' HTTP request methods is >>> one area where Django support is a little bit patchy.
>> Well that's good to know!
>>> This is mostly an artefact of the vast majority of browsers providing >>> exactly zero support for requests other that GET and POST. As a result >>> Django is (by necessity) really good at handling GET and POST; PUT, >>> DELETE et al are all essential parts of the full HTTP spec, but >>> support in Django isn't quite as comprehensive.
>>> This is one of those areas where patches are definitely welcome.
>> I'm happy to provide some patches - presumably a bug report + patch is the >> way to go.
> As always.
>> Which raises the question of what the correct behaviour should be, so the >> bug can report that it's not happening and the patch can implement.
>> As a first pass:
>> "On a PUT request, Django should populate request.PUT and request.FILES in >> the same way as for POST requests. FILES will be initialised if the content >> is multipart, and the PUT dictionary will have key/value pairs if the >> uploaded content included url-encoded form entries.
>> If the format of the datastream is not understood, it will be left in >> _raw_post_data as previously, and req.FILES and req.PUT initialised to empty >> hashes."
>> Does this sound reasonable as a bug/feature description?
> Sounds reasonable to me. I can't think of any obvious reason that the > handling for PUT should differ from POST.
> Yours, > Russ Magee %-)
Does that mean the request object could have arbitrary HTTP_METHOD attributes when it can parse the content as keys:values? At least if the MIME of the request is `application/x-www-form-urlencoded`?
I mean as far as I know nothing forbids creating custom HTTP methods (indeed, that's what e.g. WEBDAV does) and some of these custom methods could very well have the same properties as POST.
On Mon, Jan 18, 2010 at 9:23 AM, Masklinn <maskl...@masklinn.net> wrote: > On 18 Jan 2010, at 03:04 , Russell Keith-Magee wrote:
> > On Mon, Jan 18, 2010 at 8:14 AM, Malcolm Box <malcolm....@gmail.com> > wrote: > >> On Sat, Jan 16, 2010 at 4:58 AM, Russell Keith-Magee > >> Which raises the question of what the correct behaviour should be, so > the > >> bug can report that it's not happening and the patch can implement.
> >> As a first pass:
> >> "On a PUT request, Django should populate request.PUT and request.FILES > in > >> the same way as for POST requests. FILES will be initialised if the > content > >> is multipart, and the PUT dictionary will have key/value pairs if the > >> uploaded content included url-encoded form entries.
> >> If the format of the datastream is not understood, it will be left in > >> _raw_post_data as previously, and req.FILES and req.PUT initialised to > empty > >> hashes."
> >> Does this sound reasonable as a bug/feature description?
> > Sounds reasonable to me. I can't think of any obvious reason that the > > handling for PUT should differ from POST.
> Does that mean the request object could have arbitrary HTTP_METHOD > attributes when it can parse the content as keys:values? At least if the > MIME of the request is `application/x-www-form-urlencoded`?
> I mean as far as I know nothing forbids creating custom HTTP methods > (indeed, that's what e.g. WEBDAV does) and some of these custom methods > could very well have the same properties as POST.
I don't see why not, although that's not what I'm planning to submit patches for - I'm just aiming to cover PUT ie to complete the support for the core HTTP methods that are allowed entity bodies.
So the HttpRequest object will grow a new PUT attribute, but nothing else.
On Mon, Jan 18, 2010 at 5:23 PM, Masklinn <maskl...@masklinn.net> wrote: > On 18 Jan 2010, at 03:04 , Russell Keith-Magee wrote:
>> On Mon, Jan 18, 2010 at 8:14 AM, Malcolm Box <malcolm....@gmail.com> wrote: >>> On Sat, Jan 16, 2010 at 4:58 AM, Russell Keith-Magee >>> <freakboy3...@gmail.com> wrote:
>>>> On Sat, Jan 16, 2010 at 6:32 AM, Malcolm Box <malcolm....@gmail.com> >>>> wrote: >>>> <snip> >>>>> It seems to me that Django should process POST and PUT requests the same >>>>> - >>>>> i.e. the request.FILES attribute should be initialised for either if a >>>>> multipart content-type is detected.
>>>>> Have I fundamentally misunderstood how this stuff should work?
>>>> On first inspection, I don't think you've got anything fundamentally >>>> wrong. Support for PUT and the other 'exotic' HTTP request methods is >>>> one area where Django support is a little bit patchy.
>>> Well that's good to know!
>>>> This is mostly an artefact of the vast majority of browsers providing >>>> exactly zero support for requests other that GET and POST. As a result >>>> Django is (by necessity) really good at handling GET and POST; PUT, >>>> DELETE et al are all essential parts of the full HTTP spec, but >>>> support in Django isn't quite as comprehensive.
>>>> This is one of those areas where patches are definitely welcome.
>>> I'm happy to provide some patches - presumably a bug report + patch is the >>> way to go.
>> As always.
>>> Which raises the question of what the correct behaviour should be, so the >>> bug can report that it's not happening and the patch can implement.
>>> As a first pass:
>>> "On a PUT request, Django should populate request.PUT and request.FILES in >>> the same way as for POST requests. FILES will be initialised if the content >>> is multipart, and the PUT dictionary will have key/value pairs if the >>> uploaded content included url-encoded form entries.
>>> If the format of the datastream is not understood, it will be left in >>> _raw_post_data as previously, and req.FILES and req.PUT initialised to empty >>> hashes."
>>> Does this sound reasonable as a bug/feature description?
>> Sounds reasonable to me. I can't think of any obvious reason that the >> handling for PUT should differ from POST.
>> Yours, >> Russ Magee %-)
> Does that mean the request object could have arbitrary HTTP_METHOD attributes when it can parse the content as keys:values? At least if the MIME of the request is `application/x-www-form-urlencoded`?
> I mean as far as I know nothing forbids creating custom HTTP methods (indeed, that's what e.g. WEBDAV does) and some of these custom methods could very well have the same properties as POST.
FYI, Section 5.1.1 of RFC 2616 specifically allows the definition of extension methods.
I'm open to suggestions on how Django's handling of arbitrary extension methods can be improved. As it stands, request.REQUEST will contain all the query arguments that have been provided in a request, regardless of the request method. This can already be used to support extension request methods.
If you're looking for something more than this, feel free to make a specific proposal. Personally I'm not sure I agree that having request.FOOBAR is a good idea. There are many names that are legal request methods, but aren't legal variable names, so we would need to have a mapping scheme. We would also end up with a situation where extension methods would only produce attributes when used, so hasatttr(request, 'POST') would always succeed, but hasattr(request, 'FOOBAR') would only succeed if request.method == 'FOOBAR'.
However, as Malcolm notes - this is a separate issue to handling FILES on a PUT request, so it should be handled as a separate ticket and mailing list thread.
