Re: [Django] #34930: Parallel tests fail on Python 3.11+ and MacOS.
Django
-------------------------------------+-------------------------------------
Reporter: Matt Hegarty | Owner: nobody
Type: Bug | Status: closed
Component: Testing framework | Version: 4.2
Severity: Normal | Resolution: needsinfo
Keywords: pickle | Triage Stage:
_contextvars.Context _contextvars | Unreviewed
Context |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Matt Hegarty):
- Reproducible on Ubuntu and MacOS
- using python 3.11 or 3.12 (3.10 is ok)
- using Django 4.1 or higher (4.0.10 is ok)
- Needs to have 2 or more test classes
- 1 class has to subclass IsolatedAsyncioTestCase
This will show the issue:
{{{
git clone g...@github.com:matthewhegarty/tutorial.git
cd tutorial
mkvirtualenv -p `which python3.11` -r requirements.txt tutorial-311
# fails
./manage.py test --parallel
# ok
./manage.py test
}}}
Some other testing
- Django 4.0.10 [OK]
- Django 4.1 [FAIL] (but with different error)
- Django 4.1.1 [FAIL] (error from
https://code.djangoproject.com/ticket/34010)
- Django 4.1.2 [FAIL] (first instance of this error)
- Django 4.1.8 [FAIL]
--
Ticket URL: <https://code.djangoproject.com/ticket/34930#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: Matt Hegarty | Owner: nobody
Component: Testing framework | Version: 4.2
Severity: Normal | Resolution:
_contextvars.Context _contextvars | Unreviewed
Context |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: closed => new
* resolution: needsinfo =>
--
Ticket URL: <https://code.djangoproject.com/ticket/34930#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: Matt Hegarty | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 4.2
Severity: Normal | Resolution:
Keywords: pickle | Triage Stage:
_contextvars.Context _contextvars | Unreviewed
Context |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Matt Hegarty):
* Attachment "django_4_1_0_issue.txt" added.
stack trace from 4.0.1
--
Ticket URL: <https://code.djangoproject.com/ticket/34930>
Django
-------------------------------------+-------------------------------------
Reporter: Matt Hegarty | Owner: nobody
Component: Testing framework | Version: 4.2
Keywords: pickle | Triage Stage:
_contextvars.Context _contextvars | Unreviewed
Context |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => needsinfo
Comment:
I can confirm I also get the reported error with the given reproducer,
running on a Manjaro system (amd64):
{{{
Traceback (most recent call last):
File "/home/nessita/fellowship/django/tests/runtests.py", line 783, in
<module>
failures = django_tests(
^^^^^^^^^^^^^
File "/home/nessita/fellowship/django/tests/runtests.py", line 421, in
django_tests
failures = test_runner.run_tests(test_labels)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nessita/fellowship/django/django/test/runner.py", line 1068,
in run_tests
result = self.run_suite(suite)
^^^^^^^^^^^^^^^^^^^^^
File "/home/nessita/fellowship/django/django/test/runner.py", line 995,
in run_suite
return runner.run(suite)
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/unittest/runner.py", line 217, in run
test(result)
File "/usr/lib/python3.11/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nessita/fellowship/django/django/test/runner.py", line 541,
in run
subsuite_index, events = test_results.next(timeout=0.1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/pool.py", line 873, in next
raise value
File "/usr/lib/python3.11/multiprocessing/pool.py", line 540, in
_handle_tasks
put(task)
File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in
send
self._send_bytes(_ForkingPickler.dumps(obj))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/reduction.py", line 51, in
dumps
cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_contextvars.Context' object
Exception ignored in: <function Pool.__del__ at 0x7f16d6958680>
Traceback (most recent call last):
File "/usr/lib/python3.11/multiprocessing/pool.py", line 268, in __del__
ResourceWarning: unclosed running multiprocessing pool
<multiprocessing.pool.Pool state=RUN pool_size=4>
}}}
What I don't have clarity is whether this ever worked, I have bisected up
to the adding of ASGI support (commit
`a415ce70bef6d91036b00dd2c8544aed7aeeaaed`) and the given tests are still
failing. I have also checked out `stable/4.0.x` and the tests are still
not working.
Matt, can you please explain how Django is at fault here? It seems that
`IsolatedAsyncioTestCase` hasn't been (ever?) supported?
--
Ticket URL: <https://code.djangoproject.com/ticket/34930#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: Matt Hegarty | Owner: nobody
Type: Bug | Status: closed
Component: Testing framework | Version: 4.2
Severity: Normal | Resolution: needsinfo
Keywords: pickle | Triage Stage:
_contextvars.Context _contextvars | Unreviewed
Context |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Matt Hegarty):
> I have also checked out stable/4.0.x and the tests are still not
working.
- The tests pass for me when running against `stable/4.0.x` (python 3.11)
(i.e. no crash)
- I have updated the
[https://github.com/matthewhegarty/tutorial/commit/640533b5703d93f30aa8a3fb96abcf5e3a3045be
test project] to see if you can reproduce
--
Ticket URL: <https://code.djangoproject.com/ticket/34930#comment:5>