#37397: request.body does not exclude file upload data from
DATA_UPLOAD_MAX_MEMORY_SIZE, contrary to docs
---------------------------+-----------------------------------------
Reporter: marcapdev | Type: Bug
Status: new | Component: Documentation
Version: 6.1 | Severity: Normal
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------+-----------------------------------------
== Summary ==
The documentation for `DATA_UPLOAD_MAX_MEMORY_SIZE` states that the size
check, when triggered via ''either'' `request.body` or `request.POST`,
excludes file upload data (`request.FILES`) from the calculation. This is
true for `request.POST` (via `MultiPartParser`), but is '''not true''' for
`request.body` — that property compares the full `Content-Length` header
(file bytes included) against the setting, with no file-exclusion logic at
all, and has nothing to do with `request.FILES`.
== Documentation text (verbatim,
https://docs.djangoproject.com/en/6.1/ref/settings/#data-upload-max-
memory-size) ==
The maximum size in bytes that a request body may be before a
`SuspiciousOperation` (`RequestDataTooBig`) is raised. '''The check is
done when accessing `request.body` or `request.POST` and is calculated
against the total request size excluding any file upload data
(`request.FILES`).''' You can set this to `None` to disable the check.
Applications that are expected to receive unusually large form posts
should tune this setting.
== What the code actually does (tag 6.1.1, current stable) ==
'''`request.body`''' (`django/http/request.py:399-442`):
{{{#!python
@property
def body(self):
if not hasattr(self, "_body"):
if self._read_started:
raise RawPostDataException(
"You cannot access body after reading from request's data
stream"
)
# Make Content-Length fall back to 0 if malformed (e.g.
ASGIRequest
# comma-joins duplicate Content-Length headers).
try:
content_length = int(self.META.get("CONTENT_LENGTH") or 0)
except (ValueError, TypeError):
content_length = 0
# Limit the maximum request data size that will be handled
# in-memory. Reject early when Content-Length is present and
# already exceeds the limit, avoiding reading the body at all.
self._check_data_too_big(content_length)
# Content-Length can be absent or understated (e.g.
# `Transfer-Encoding: chunked` on ASGI), so for seekable
# streams (e.g. SpooledTemporaryFile on ASGI), check the actual
# buffered size before reading it all into memory.
if self._stream.seekable():
stream_size = self._stream.seek(0, os.SEEK_END)
self._check_data_too_big(stream_size)
self._stream.seek(0)
try:
self._body = self.read()
except OSError as e:
raise UnreadablePostError(*e.args) from e
finally:
self._stream.close()
self._stream = BytesIO(self._body)
return self._body
def _check_data_too_big(self, length):
if (
settings.DATA_UPLOAD_MAX_MEMORY_SIZE is not None
and length > settings.DATA_UPLOAD_MAX_MEMORY_SIZE
):
raise RequestDataTooBig(
"Request body exceeded settings.DATA_UPLOAD_MAX_MEMORY_SIZE."
)
}}}
'''`request.POST`''' (for multipart requests, via
`django/http/multipartparser.py:222-256`):
The parser only accumulates bytes toward the limit for `FIELD`-type parts:
{{{#!python
if item_type == FIELD:
# Avoid reading more than DATA_UPLOAD_MAX_MEMORY_SIZE.
if settings.DATA_UPLOAD_MAX_MEMORY_SIZE is not None:
read_size = settings.DATA_UPLOAD_MAX_MEMORY_SIZE - num_bytes_read
...
data = field_stream.read(size=read_size)
num_bytes_read += len(data)
...
if (
settings.DATA_UPLOAD_MAX_MEMORY_SIZE is not None
and num_bytes_read > settings.DATA_UPLOAD_MAX_MEMORY_SIZE
):
raise RequestDataTooBig(
"Request body exceeded settings.DATA_UPLOAD_MAX_MEMORY_SIZE."
)
elif item_type == FILE:
# Avoid storing more than DATA_UPLOAD_MAX_NUMBER_FILES.
...
# routed to upload handlers instead; num_bytes_read is never touched
here
}}}
`FILE`-type parts are routed to upload handlers instead and never touch
`num_bytes_read` / this check at all. So for `request.POST`, the
documentation's claim is accurate — file content genuinely is excluded,
and this is the code path that actually populates `request.FILES`.
== Why this matters in practice ==
Code that accesses `request.body` for a reason unrelated to parsing form
data (e.g. logging/backing up the raw body for error reporting, signature
verification helpers, etc.) on a multipart file-upload request will
unexpectedly raise `RequestDataTooBig` for files well under what a
developer would reasonably expect from reading this documentation — since
the documented behavior ("excludes file upload data (`request.FILES`)")
simply does not apply to that code path. We hit this in production: a
multipart request with a single ~5 MB file (default
`DATA_UPLOAD_MAX_MEMORY_SIZE` of 2.5 MB) crashed with `RequestDataTooBig`
the moment a piece of unrelated middleware touched `request.body` before
the view accessed `request.POST`/`request.FILES` — despite the documented
guarantee that file data doesn't count toward this limit.
== Suggested fix ==
Clarify the documentation to note the distinction explicitly, e.g.:
The check is done when accessing `request.body` or `request.POST`. For
`request.POST` (i.e. parsed multipart form data), the calculation excludes
any file upload data (`request.FILES`). For `request.body`, the check is
performed against the full size of the request body, '''including any file
upload data''' — since `request.body` materializes the entire request body
in memory (or verifies its buffered size) regardless of its content, the
size limit applies to the whole body in that case.
== Reproduction ==
1. Set `DATA_UPLOAD_MAX_MEMORY_SIZE` to its default (2.5 MB, or any value
< file size).
2. Add trivial middleware: `def process_request(self, request):
request.body` (or anything that accesses `.body` before the view touches
`.POST`/`.FILES`).
3. POST a multipart/form-data request with a single file field larger
than the limit (and no other large fields) to any view.
4. Observe `RequestDataTooBig: Request body exceeded
settings.DATA_UPLOAD_MAX_MEMORY_SIZE.` is raised — even though the file is
the only thing contributing meaningfully to the request size, and the
documentation states file data is excluded from this check.
--
Ticket URL: <
https://code.djangoproject.com/ticket/37397>
Django <
https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.