[Django] #35010: Parallel test runner hangs if one of the tests segfault

26 views
Skip to first unread message

Django

unread,
Dec 4, 2023, 10:23:42 AM12/4/23
to django-...@googlegroups.com
#35010: Parallel test runner hangs if one of the tests segfault
---------------------------------------------+------------------------
Reporter: Michał Górny | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: dev
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
When the tests are being run in parallel, and one of the tests cause a
segfault, the test runner hangs:

{{{
$ PYTHONPATH=. python3.11 tests/runtests.py --parallel=2
queries.tests.{ProxyQueryCleanupTest,Queries1Tests}
Testing against Django installed in '/tmp/django/django' with up to 2
processes
Found 78 test(s).
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
System check identified no issues (1 silenced).
.Fatal Python error: Segmentation fault

Current thread 0x00007fe3c3d42740 (most recent call first):
File "/usr/lib/python3.11/copy.py", line 265 in _reconstruct
File "/usr/lib/python3.11/copy.py", line 102 in copy
File "/tmp/django/django/db/models/expressions.py", line 426 in copy
File "/tmp/django/django/db/models/expressions.py", line 284 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 277 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
File "/tmp/django/django/db/models/sql/where.py", line 279 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 275 in
_resolve_node
File "/tmp/django/django/db/models/sql/where.py", line 283 in
resolve_expression
File "/tmp/django/django/db/models/sql/query.py", line 1203 in
resolve_expression
File "/tmp/django/django/db/models/sql/where.py", line 268 in
_resolve_leaf
...

}}}

At this point, nothing else is happening, the processes don't utilize CPU:

{{{
$ ps -ft
111241 pts/2 Tl 0:02 \_ python3.11 tests/runtests.py
--parallel=2 queries.tests.ProxyQueryCleanupTest
queries.tests.Queries1Tests
111243 pts/2 T 0:00 | \_ python3.11 tests/runtests.py
--parallel=2 queries.tests.ProxyQueryCleanupTest
queries.tests.Queries1Tests
111248 pts/2 T 0:00 | \_ python3.11 tests/runtests.py
--parallel=2 queries.tests.ProxyQueryCleanupTest
queries.tests.Queries1Tests
}}}

If I cancel it, it seems to ignore the killed process:

{{{
^C
----------------------------------------------------------------------
Ran 1 test in 613.663s

OK
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
}}}

(I'm still trying to figure out why that other test segfaults for me)

--
Ticket URL: <https://code.djangoproject.com/ticket/35010>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,
Dec 4, 2023, 10:33:29 AM12/4/23
to django-...@googlegroups.com
#35010: Parallel test runner hangs if one of the tests segfault
-----------------------------------+--------------------------------------

Reporter: Michał Górny | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Michał Górny):

As a note, the segfault is apparently caused by `jedi` setting a high
recursion limit globally (sigh), and it gets imported via `ipdb`.

--
Ticket URL: <https://code.djangoproject.com/ticket/35010#comment:1>

Django

unread,
Dec 4, 2023, 9:38:30 PM12/4/23
to django-...@googlegroups.com
#35010: Parallel test runner hangs if one of the tests segfault
-----------------------------------+--------------------------------------
Reporter: Michał Górny | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Simon Charette):

I'm afraid this will be very hard to solve
[https://bugs.python.org/issue22393 as this is bug]
`multiprocessing.Pool`.

A solution alluded to in the Python bug report is to use
`concurrent.futures.ProcessPoolExecutor` instead as it doesn't seem to be
affected by this problem.

Would you be interested in trying to
[https://github.com/django/django/blob/14884b6be89963c8ceb2c9da459ed8421aea3bf8/django/test/runner.py#L516-L528
adapt the current solution]?

--
Ticket URL: <https://code.djangoproject.com/ticket/35010#comment:2>

Django

unread,
Dec 4, 2023, 11:57:10 PM12/4/23
to django-...@googlegroups.com
#35010: Parallel test runner hangs if one of the tests segfault
-----------------------------------+--------------------------------------
Reporter: Michał Górny | Owner: nobody
Type: Bug | Status: closed

Component: Testing framework | Version: dev
Severity: Normal | Resolution: invalid

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------
Changes (by Mariusz Felisiak):

* status: new => closed
* resolution: => invalid


Comment:

Thanks for the report and investigation, however I don't think it's
something we should workaround in Django itself.

Marking as "invalid" because this is an issue in Python.

--
Ticket URL: <https://code.djangoproject.com/ticket/35010#comment:3>

Reply all
Reply to author
Forward
0 new messages