[Django] #36770: SQLite threading tests are flaky when parallel test suite runs in forkserver/spawn

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver/spawn
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Type:
| Cleanup/optimization
Status: new | Component: Testing
| framework
Version: 5.2 | Severity: Normal
Keywords: 3.14, forkserver, | Triage Stage:
spawn, parallel | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
We have two tests often failing on GitHub Actions CI runs under the
parallel test runner having to do with threading and sqlite in-memory
databases.

- `backends.sqlite.tests.ThreadSharing.test_database_sharing_in_threads`
-
`servers.tests.LiveServerInMemoryDatabaseLockTest.test_in_memory_database_lock`

As of now, the failures are most common on the byte-compiled Django
workflow, but we've at least seen the `test_in_memory_database_lock`
failure on other workflows.

Tracebacks:

{{{#!py
======================================================================
FAIL: test_database_sharing_in_threads
(backends.sqlite.tests.ThreadSharing.test_database_sharing_in_threads)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py",
line 58, in testPartExecutor
yield
File
"/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py",
line 669, in run
self._callTestMethod(testMethod)

File
"/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py",
line 615, in _callTestMethod
result = method()
^^^^^^^^^^^^^^^
File "/home/runner/work/django/django/tests/backends/sqlite/tests.py",
line 282, in test_database_sharing_in_threads
self.assertEqual(Object.objects.count(), 2)
^^^^^^^^^^^
File
"/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py",
line 925, in assertEqual
assertion_func(first, second, msg=msg)
^^^^^^^^^^^^^^^
File
"/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py",
line 918, in _baseAssertEqual
raise self.failureException(msg)
^^^^^^^^^^^
AssertionError: 1 != 2

----------------------------------------------------------------------
}}}

{{{#!py
test_in_memory_database_lock
(servers.tests.LiveServerInMemoryDatabaseLockTest.test_in_memory_database_lock)
failed:

AssertionError('Unexpected error due to a database lock.')
}}}

Other times, the workflow deadlocks, so we don't know which test failed,
which caused us to add `timeout-minutes: 60` everywhere defensively in
e48527f91d341c85a652499a5baaf725d36ae54f.

This failure started manifesting after we upgraded more CI jobs to Python
3.14, which defaults POSIX systems to the `forkserver` multiprocessing
mode. See #36531.

I haven't been successful reproducing locally.

I have a low-confidence hypothesis that it might be something do with
calls to `setup_worker_connection` inside `_init_worker` that occur during
the middle of the test runs when there are resource contentions. Is it
possible that a late worker init is clobbering some of these particular
tests' setup where database connections are being overwritten to be the
same (?)
--
Ticket URL: <https://code.djangoproject.com/ticket/36770>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver/spawn
-------------------------------------+-------------------------------------

Reporter: Jacob Walls | Owner: (none)
Type: | Status: new
Cleanup/optimization |

Component: Testing framework | Version: 5.2

Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
spawn, parallel |

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* stage: Unreviewed => Accepted

Comment:

Thank you!
--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:1>