Re: [Django] #35838: Support Requests with Transfer-Encoding: Chunked

[Django]

#35838: Support Requests with Transfer-Encoding: Chunked
------------------------------------+------------------------------------
Reporter: Klaas van Schelven | Owner: (none)
Type: New feature | Status: new
Component: HTTP handling | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+------------------------------------
Comment (by Carlton Gibson):

> …add Transfer-Encoding: Chunked support more generally and mark this as
a duplicate.

They're certainly both closely related. Django receives the de-chunked
request, but then doesn't handle it as expected (because of the missing
Content Length header).

> ...support more generally...

Given that both WSGI and ASGI specify that the server de-chunks the
request, I'm not sure what else would be needed here? 🤔
--
Ticket URL: <https://code.djangoproject.com/ticket/35838#comment:13>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35838: Support Requests with Transfer-Encoding: Chunked
------------------------------------+------------------------------------
Reporter: Klaas van Schelven | Owner: (none)
Type: New feature | Status: new
Component: HTTP handling | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+------------------------------------

Comment (by bcail):

> Given that both WSGI and ASGI specify that the server de-chunks the
request

What does it mean to "de-chunk" the request? Is there a description
somewhere I could look at?
--
Ticket URL: <https://code.djangoproject.com/ticket/35838#comment:14>

[Django]

#35838: Support Requests with Transfer-Encoding: Chunked
------------------------------------+------------------------------------
Reporter: Klaas van Schelven | Owner: (none)
Type: New feature | Status: new
Component: HTTP handling | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+------------------------------------

Comment (by Klaas van Schelven):

> Given that both WSGI and ASGI specify that the server de-chunks the

request, I'm not sure what else would be needed here?

whatever the precise definition (spec) that the first part of that
sentence implies... "what else would be needed here" can be (and was)
answered by me as "don't silently throw away request bodies when Content-
Length is missing"
--
Ticket URL: <https://code.djangoproject.com/ticket/35838#comment:15>

[Django]

#35838: Support Requests with Transfer-Encoding: Chunked
------------------------------------+------------------------------------
Reporter: Klaas van Schelven | Owner: (none)
Type: New feature | Status: new
Component: HTTP handling | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+------------------------------------

Comment (by bcail):

It seems like "de-chunk the request" is another way of saying "decoding
any inbound Transfer-Encoding, including chunked encoding if applicable"
(from WSGI [https://peps.python.org/pep-3333/#other-http-features
PEP3333]). RFC2616 describes how to
[https://datatracker.ietf.org/doc/html/rfc2616.html#section-19.4.6 decode
a chunked request]:
{{{
length := 0
read chunk-size, chunk-extension (if any) and CRLF
while (chunk-size > 0) {
read chunk-data and CRLF
append chunk-data to entity-body
length := length + chunk-size
read chunk-size and CRLF
}
read entity-header
while (entity-header not empty) {
append entity-header to existing header fields
read entity-header
}
Content-Length := length
Remove "chunked" from Transfer-Encoding
}}}
This looks to me like the WSGI server should set the Content-Length
header, and remove "chunked" from Transfer-Encoding, before passing the
request to Django (unless Django and the WSGI servers decide to ignore the
spec here).
--
Ticket URL: <https://code.djangoproject.com/ticket/35838#comment:16>

[Django]

#35838: Support Requests with Transfer-Encoding: Chunked
------------------------------------+------------------------------------
Reporter: Klaas van Schelven | Owner: (none)
Type: New feature | Status: new
Component: HTTP handling | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+------------------------------------

Comment (by Klaas van Schelven):

> RFC2616 describes how to ​decode a chunked request:

> This looks to me like the WSGI server should set the Content-Length
header

That is indeed one interpretation, but I'm not sure it's the only valid
one. In the quoted RFC Content-Length is calculated, but the RFC does not
say this should be translated into a header by a wsgi server (obviously,
since the RFC doesn't deal with WSGI).
--
Ticket URL: <https://code.djangoproject.com/ticket/35838#comment:17>

[Django]

#35838: Support Requests with Transfer-Encoding: Chunked
------------------------------------+------------------------------------
Reporter: Klaas van Schelven | Owner: (none)
Type: New feature | Status: new
Component: HTTP handling | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+------------------------------------

Changes (by Simon Brulhart):

* cc: Simon Brulhart (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/35838#comment:18>

[Django]

#35838: Support Requests with Transfer-Encoding: Chunked
------------------------------------+------------------------------------
Reporter: Klaas van Schelven | Owner: (none)
Type: New feature | Status: new
Component: HTTP handling | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+------------------------------------

Changes (by Simon Charette):

* cc: Simon Charette (added)

Comment:

We ran into issue at work today as we're moving away from using Nginx for
some services and it happened to handles de-chunking by default which
means that by the time the request reached the WSGI compatible server
(gunicorn in our case) it already had a `Content-Length` header.
--
Ticket URL: <https://code.djangoproject.com/ticket/35838#comment:19>

[Django]

#35838: Support Requests with Transfer-Encoding: Chunked
------------------------------------+------------------------------------

Reporter: Klaas van Schelven | Owner: Pravin
Type: New feature | Status: assigned

Component: HTTP handling | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+------------------------------------

Changes (by Pravin):

* owner: (none) => Pravin
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/35838#comment:20>

[Django]

#35838: Support Requests with Transfer-Encoding: Chunked
------------------------------------+------------------------------------
Reporter: Klaas van Schelven | Owner: Pravin
Type: New feature | Status: assigned
Component: HTTP handling | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+------------------------------------
Changes (by Pravin):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35838#comment:21>