[Django] #36056: Fix ignored exceptions in OutputWrapper.flush()
13 views
Skip to first unread message
Django
Jan 2, 2025, 5:57:18 AMJan 2
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Type: Bug
Status: new | Component: Core
| (Management commands)
Version: dev | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Running pytest on two different projects with Python 3.13, I’ve seen this
error logged after tests pass:
{{{
$ pytest
...
==== 1 passed in 0.01s ====
Exception ignored in: <django.core.management.base.OutputWrapper object at
0x173326230>
Traceback (most recent call last):
File "/.../.venv/lib/python3.13/site-
packages/django/core/management/base.py", line 171, in flush
self._out.flush()
ValueError: I/O operation on closed file.
...
}}}
It can appear multiple times, depending on how many management commands
are tested.
This message is from Python’s
[https://docs.python.org/3.12/library/sys.html#sys.unraisablehook
unraisable exception hook], logging the minimal available information for
an exception raised during a destructor or garbage collection.
I minimized the test case to find that it requires a test that captures
the exception from a failing management command, minimized to:
{{{#!python
import pytest
from django.core.management import call_command
from django.core.management.base import CommandError
from django.test import SimpleTestCase
class ExampleTests(SimpleTestCase):
def test_it(self):
with pytest.raises(CommandError) as excinfo:
call_command("check", "--non")
}}}
This test runs the `check` management command with a non-existent option,
triggering an error.
It can be run with `pytest` and `pytest-django` installed like:
{{{
$ pytest --ds=test_example test_example.py
========================= test session starts =========================
platform darwin -- Python 3.13.1, pytest-8.3.4, pluggy-1.5.0
django: version: 5.1.4, settings: test_example (from option)
rootdir: /.../example
configfile: pyproject.toml
plugins: cov-6.0.0, django-4.9.0
collected 1 item
test_example.py . [100%]
========================== 1 passed in 0.01s ==========================
Exception ignored in: <django.core.management.base.OutputWrapper object at
0x10533b760>
Traceback (most recent call last):
File "/.../.venv/lib/python3.13/site-
packages/django/core/management/base.py", line 171, in flush
self._out.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <django.core.management.base.OutputWrapper object at
0x10549b760>
Traceback (most recent call last):
File "/.../.venv/lib/python3.13/site-
packages/django/core/management/base.py", line 171, in flush
self._out.flush()
ValueError: I/O operation on closed file.
}}}
(`-ds=test_example` specifies the Django settings module as the test
module, so you don’t need a whole project set up.)
After some drilling down, I determined the cause is `OutputWrapper`
instances that are created during `BaseCommand.__init__` with references
to `sys.stdout` and `sys.stderr`. It seems that when they’re deleted, they
interact poorly if the underlying file objects have already been closed.
pytest’s output caputring installs a mock `sys.stdout` and closes it after
tests are done, but the captured traceback retains a reference to the
`OutputWrapper` until tests finish. When the `OutputWrapper` is finally
garbage collected, the error is logged. Disabling output capturing with
`pytest -s` makes the error go away.
I tried to replicate within Django’s test framework with its output
capturing option, `--buffer`, but it didn’t work. I think it has some
differences that mean the error doesn’t appear.
But I did manage to minimize down to this test script, free of pytest and
management commands:
{{{#!python
import io
from django.core.management.base import OutputWrapper
out = io.TextIOWrapper(io.BytesIO())
wrapper = OutputWrapper(out)
out.close()
}}}
Running it shows:
{{{
$ python example_simplest.py
Exception ignored in: <django.core.management.base.OutputWrapper object at
0x1049cbb20>
Traceback (most recent call last):
File "/.../.venv/lib/python3.13/site-
packages/django/core/management/base.py", line 172, in flush
ValueError: I/O operation on closed file.
}}}
I also found that the error is reproducible on Python < 3.13 when enabling
development mode with `python -X dev`. It turns out the logging was
special-cased to development mode within the io module, but this special-
casing was removed in Python 3.13 in
[https://github.com/python/cpython/commit/58a2e0981642dcddf49daa776ff68a43d3498cee
commit 58a2e0981642dcddf49daa776ff68a43d3498cee]. So the error has been
there all along, but now it’s more visible.
I think the fix is to stop `OutputWrapper` from inheriting from
`TextIOBase`, as was added in dc8834cad41aa407f402dc54788df3cd37ab3e22.
That commit was focused on removing `force_str()` calls and it’s not clear
why the inheritance was added. But it seems that the path
[https://github.com/python/cpython/blob/e1baa778f602ede66831eb34b9ef17f21e4d4347/Lib/_pyio.py#L397
from `IOBase.__del__()`] to the custom `flush()` is what’s causing the
error.
--
Ticket URL: <https://code.djangoproject.com/ticket/36056>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Type: Bug
Status: new | Component: Core
| (Management commands)
Version: dev | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Running pytest on two different projects with Python 3.13, I’ve seen this
error logged after tests pass:
{{{
$ pytest
...
==== 1 passed in 0.01s ====
Exception ignored in: <django.core.management.base.OutputWrapper object at
0x173326230>
Traceback (most recent call last):
File "/.../.venv/lib/python3.13/site-
packages/django/core/management/base.py", line 171, in flush
self._out.flush()
ValueError: I/O operation on closed file.
...
}}}
It can appear multiple times, depending on how many management commands
are tested.
This message is from Python’s
[https://docs.python.org/3.12/library/sys.html#sys.unraisablehook
unraisable exception hook], logging the minimal available information for
an exception raised during a destructor or garbage collection.
I minimized the test case to find that it requires a test that captures
the exception from a failing management command, minimized to:
{{{#!python
import pytest
from django.core.management import call_command
from django.core.management.base import CommandError
from django.test import SimpleTestCase
class ExampleTests(SimpleTestCase):
def test_it(self):
with pytest.raises(CommandError) as excinfo:
call_command("check", "--non")
}}}
This test runs the `check` management command with a non-existent option,
triggering an error.
It can be run with `pytest` and `pytest-django` installed like:
{{{
$ pytest --ds=test_example test_example.py
========================= test session starts =========================
platform darwin -- Python 3.13.1, pytest-8.3.4, pluggy-1.5.0
django: version: 5.1.4, settings: test_example (from option)
rootdir: /.../example
configfile: pyproject.toml
plugins: cov-6.0.0, django-4.9.0
collected 1 item
test_example.py . [100%]
========================== 1 passed in 0.01s ==========================
Exception ignored in: <django.core.management.base.OutputWrapper object at
0x10533b760>
Traceback (most recent call last):
File "/.../.venv/lib/python3.13/site-
packages/django/core/management/base.py", line 171, in flush
self._out.flush()
ValueError: I/O operation on closed file.
Exception ignored in: <django.core.management.base.OutputWrapper object at
0x10549b760>
Traceback (most recent call last):
File "/.../.venv/lib/python3.13/site-
packages/django/core/management/base.py", line 171, in flush
self._out.flush()
ValueError: I/O operation on closed file.
}}}
(`-ds=test_example` specifies the Django settings module as the test
module, so you don’t need a whole project set up.)
After some drilling down, I determined the cause is `OutputWrapper`
instances that are created during `BaseCommand.__init__` with references
to `sys.stdout` and `sys.stderr`. It seems that when they’re deleted, they
interact poorly if the underlying file objects have already been closed.
pytest’s output caputring installs a mock `sys.stdout` and closes it after
tests are done, but the captured traceback retains a reference to the
`OutputWrapper` until tests finish. When the `OutputWrapper` is finally
garbage collected, the error is logged. Disabling output capturing with
`pytest -s` makes the error go away.
I tried to replicate within Django’s test framework with its output
capturing option, `--buffer`, but it didn’t work. I think it has some
differences that mean the error doesn’t appear.
But I did manage to minimize down to this test script, free of pytest and
management commands:
{{{#!python
import io
from django.core.management.base import OutputWrapper
out = io.TextIOWrapper(io.BytesIO())
wrapper = OutputWrapper(out)
out.close()
}}}
Running it shows:
{{{
$ python example_simplest.py
Exception ignored in: <django.core.management.base.OutputWrapper object at
0x1049cbb20>
Traceback (most recent call last):
File "/.../.venv/lib/python3.13/site-
packages/django/core/management/base.py", line 172, in flush
ValueError: I/O operation on closed file.
}}}
I also found that the error is reproducible on Python < 3.13 when enabling
development mode with `python -X dev`. It turns out the logging was
special-cased to development mode within the io module, but this special-
casing was removed in Python 3.13 in
[https://github.com/python/cpython/commit/58a2e0981642dcddf49daa776ff68a43d3498cee
commit 58a2e0981642dcddf49daa776ff68a43d3498cee]. So the error has been
there all along, but now it’s more visible.
I think the fix is to stop `OutputWrapper` from inheriting from
`TextIOBase`, as was added in dc8834cad41aa407f402dc54788df3cd37ab3e22.
That commit was focused on removing `force_str()` calls and it’s not clear
why the inheritance was added. But it seems that the path
[https://github.com/python/cpython/blob/e1baa778f602ede66831eb34b9ef17f21e4d4347/Lib/_pyio.py#L397
from `IOBase.__del__()`] to the custom `flush()` is what’s causing the
error.
--
Ticket URL: <https://code.djangoproject.com/ticket/36056>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Jan 2, 2025, 6:56:23 AMJan 2
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: (none)
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Adam Johnson):
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:1>
Django
Jan 2, 2025, 6:56:35 AMJan 2
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
-------------------------------------+-------------------------------------
| Johnson
Type: Bug | Status: assigned
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Adam Johnson):
* owner: (none) => Adam Johnson
commands) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Adam Johnson):
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:2>
Django
Jan 2, 2025, 10:22:31 AMJan 2
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Cleanup/optimization | Status: assigned
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
commands) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Natalia Bidart):
-------------------------------------+-------------------------------------
* needs_tests: 0 => 1
* stage: Unreviewed => Accepted
* type: Bug => Cleanup/optimization
Comment:
Thank you Adam for the very detailed ticket report. I think I have
followed your rationale and I have reproduced the example in the Python
terminal. Accepting on the basis that we can add a test to the proposed PR
with something similar to what is proposed as the pure python script. Do
you think you could try to work a test out?
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:3>
Django
Jan 2, 2025, 10:52:11 AMJan 2
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Natalia Bidart):
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
I wonder if the fix for this shouldn't be pushing forward #21429.
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:4>
Django
Jan 2, 2025, 10:57:43 AMJan 2
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Adam Johnson):
-------------------------------------+-------------------------------------
* needs_tests: 1 => 0
Comment:
Test added!
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:5>
Django
Jan 3, 2025, 10:24:05 AMJan 3
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Natalia Bidart):
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:6>
Django
Jan 3, 2025, 10:30:02 PMJan 3
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: closed
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution: fixed
commands) |
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by GitHub <noreply@…>):
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* resolution: => fixed
* status: assigned => closed
Comment:
In [changeset:"ec0e784f91b551c654f0962431cc31091926792d" ec0e784f]:
{{{#!CommitTicketReference repository=""
revision="ec0e784f91b551c654f0962431cc31091926792d"
Fixed #36056 -- Made OutputWrapper a virtual subclass of TextIOBase.
This fixes the ignored exception in self._out.flush() from
django.core.management.base.OutputWrapper:
`ValueError: I/O operation on closed file.`
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:7>
Django
Jan 6, 2025, 5:06:26 AMJan 6
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: closed
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Adam Johnson):
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: closed
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Thank you for merging.
I would like this to be backported to Django 5.1, although I can see that
it requires a little generosity interpreting
[https://docs.djangoproject.com/en/dev/internals/release-process
/#supported-versions the policy].
On Python 3.13 and pytest, *any* test running a failing management command
triggers this error. On the two projects I’ve seen, that means 10+ copies
of this exception in the output of a full test run. Also, there’s no easy
way to silence the error.
It’s an exception, and we could consider it a “Crashing bug”, given it
crashes part of garbage collection but not the full process.
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:8>
Django
Jan 6, 2025, 8:41:47 AMJan 6
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: closed
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Natalia Bidart):
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: closed
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Adam, let me ponder and discuss with other Fellows and I'll get back to
you.
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:9>
Django
Jan 7, 2025, 12:27:05 PMJan 7
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: closed
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Natalia Bidart):
Hey Adam, while I sympathize with your request, we won't backport this
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: closed
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Natalia Bidart):
fix. There is some further rationale in [https://forum.djangoproject.com/t
/backport-policy-when-a-crashing-error-is-also-a-regression/37520 this
related forum post].
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:10>
Django
Jan 8, 2025, 5:16:35 AMJan 8
to django-...@googlegroups.com
#36056: Fix ignored exceptions in OutputWrapper.flush()
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: closed
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Adam Johnson):
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: closed
Component: Core (Management | Version: dev
commands) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Okay, well it’s fair that you don’t want to assume the risk. I have
blogged about a workaround: https://adamj.eu/tech/2025/01/08/django-
silence-exception-ignored-outputwrapper/
--
Ticket URL: <https://code.djangoproject.com/ticket/36056#comment:11>
Reply all
Reply to author
Forward
0 new messages