Re: [Django] #36655: GZipMiddleware buffers streaming responses (was: GzipMiddleware buffers streaming responses)

[Django]

#36655: GZipMiddleware buffers streaming responses
-------------------------------+----------------------------------------
Reporter: Adam Johnson | Owner: Adam Johnson
Type: Bug | Status: assigned
Component: HTTP handling | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+----------------------------------------
Changes (by Adam Johnson):

* summary: GzipMiddleware buffers streaming responses => GZipMiddleware
buffers streaming responses


Old description:

> Currently, `GzipMiddleware`, via `compress_sequence()`, buffers the
> entire response before sending it to the client. This can cause issues
> for clients that expect to receive data in chunks, such as those using
> Server-Sent Events (SSE) or WebSockets.
>
> This issue was reported to me in the django-browser-reload project back
> in [https://github.com/adamchainz/django-browser-reload/pull/161 Issue
> #161 (2023)], where a contributor fixed it with a workaround, and I
> didn't think to investigate. Now, while implementing
> [https://github.com/adamchainz/django-http-compression django-http-
> compression], I have realized that it’s a proper bug that can be fixed by
> adding a call to `zfile.flush()`, as done in its
> [https://github.com/adamchainz/django-http-compression/pull/8 PR #8].
>
> To reproduce the issue, use the below app, which can be run with `uv run
> --script`. If you comment out `GzipMiddleware` and load the page in a
> browser, you will see the numbers incrementing every second. If you
> include `GzipMiddleware`, the page will never load. Adding the
> `zfile.flush()` call in `compress_sequence()` fixes the issue.
>
> {{{#!python
> #!/usr/bin/env uv run --script
> # /// script
> # requires-python = ">=3.14"
> # dependencies = [
> # "django",
> # ]
> # ///
> from __future__ import annotations
>
> import os
> import sys
> import time
>
> from django.conf import settings
> from django.core.wsgi import get_wsgi_application
> from django.http import StreamingHttpResponse
> from django.urls import path
>
> settings.configure(
> # Dangerous: disable host header validation
> ALLOWED_HOSTS=["*"],
> # Use DEBUG=1 to enable debug mode
> DEBUG=(os.environ.get("DEBUG", "") == "1"),
> # Make this module the urlconf
> ROOT_URLCONF=__name__,
> # Only gzip middleware
> MIDDLEWARE=[
> "django.middleware.gzip.GZipMiddleware",
> ],
> )
>

> def clock(request):
> def stream():
> yield "<h1>Clock</h1>\n"
> count = 1
> while True:
> yield f"<p>{count}</p>\n"
> count += 1
> time.sleep(1)
>
> return StreamingHttpResponse(stream())
>

> urlpatterns = [
> path("", clock),
> ]
>
> app = get_wsgi_application()
>
> if __name__ == "__main__":
> from django.core.management import execute_from_command_line
>
> execute_from_command_line(sys.argv)
> }}}

New description:

Currently, `GZipMiddleware`, via `compress_sequence()`, buffers the entire
response before sending it to the client. This can cause issues for
clients that expect to receive data in chunks, such as those using Server-
Sent Events (SSE) or WebSockets.

This issue was reported to me in the django-browser-reload project back in
[https://github.com/adamchainz/django-browser-reload/pull/161 Issue #161
(2023)], where a contributor fixed it with a workaround, and I didn't
think to investigate. Now, while implementing
[https://github.com/adamchainz/django-http-compression django-http-
compression], I have realized that it’s a proper bug that can be fixed by
adding a call to `zfile.flush()`, as done in its
[https://github.com/adamchainz/django-http-compression/pull/8 PR #8].

To reproduce the issue, use the app below, which can be run with `uv run
--script`. If you comment out `GzipMiddleware` and load the page in a
browser, you will see the numbers incrementing every second. If you
include `GzipMiddleware`, the page will never load. Adding the
`zfile.flush()` call in `compress_sequence()` fixes the issue.

{{{#!python
#!/usr/bin/env uv run --script
# /// script
# requires-python = ">=3.14"
# dependencies = [
# "django",
# ]
# ///
from __future__ import annotations

import os
import sys
import time

from django.conf import settings
from django.core.wsgi import get_wsgi_application
from django.http import StreamingHttpResponse
from django.urls import path

settings.configure(
# Dangerous: disable host header validation
ALLOWED_HOSTS=["*"],
# Use DEBUG=1 to enable debug mode
DEBUG=(os.environ.get("DEBUG", "") == "1"),
# Make this module the urlconf
ROOT_URLCONF=__name__,
# Only gzip middleware
MIDDLEWARE=[
"django.middleware.gzip.GZipMiddleware",
],
)


def clock(request):
def stream():
yield "<h1>Clock</h1>\n"
count = 1
while True:
yield f"<p>{count}</p>\n"
count += 1
time.sleep(1)

return StreamingHttpResponse(stream())


urlpatterns = [
path("", clock),
]

app = get_wsgi_application()

if __name__ == "__main__":
from django.core.management import execute_from_command_line

execute_from_command_line(sys.argv)
}}}

--
--
Ticket URL: <https://code.djangoproject.com/ticket/36655#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.