Re: [Django] #36770: SQLite threading tests are flaky when parallel test suite runs in forkserver mode

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver mode
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Kundan
Type: | Yadav
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
parallel |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Jacob Walls):

I've seen this intermittently locally. Leaving aside the assertion
failures for SQLite, we shouldn't have a hang in `LiveServerTestCase` when
tests fail. You can engineer a hang like this, setting a miniscule timeout
that will always raise:

{{{#!py
diff --git a/django/test/testcases.py b/django/test/testcases.py
index 5f83612fe5..622b938dd6 100644
--- a/django/test/testcases.py
+++ b/django/test/testcases.py
@@ -1844,7 +1844,8 @@ class LiveServerTestCase(TransactionTestCase):
cls.addClassCleanup(cls._terminate_thread)

# Wait for the live server to be ready
- cls.server_thread.is_ready.wait()
+ if not cls.server_thread.is_ready.wait(timeout=0.001):
+ raise Exception("Live server never became ready.")
if cls.server_thread.error:
raise cls.server_thread.error
}}}


Then when `KeyboardInterrupt`ing out of it, you get a stack trace from
`doClassCleanups`, suggesting that the termination code is waiting
forever, even though the live server never started:

{{{#!py
File
"/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/unittest/suite.py",
line 181, in _handleClassSetUp
doClassCleanups()
~~~~~~~~~~~~~~~^^
File
"/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/unittest/case.py",
line 720, in doClassCleanups
function(*args, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^^
File "/Users/jwalls/django/django/test/testcases.py", line 1864, in
_terminate_thread
cls.server_thread.terminate()
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/jwalls/django/django/test/testcases.py", line 1788, in
terminate
self.join()
}}}

Something like this fixes it:

{{{#!diff
diff --git a/django/test/testcases.py b/django/test/testcases.py
index 5f83612fe5..9cbeeeca25 100644
--- a/django/test/testcases.py
+++ b/django/test/testcases.py
@@ -1781,11 +1781,12 @@ class LiveServerThread(threading.Thread):
)

def terminate(self):
- if hasattr(self, "httpd"):
- # Stop the WSGI server
- self.httpd.shutdown()
- self.httpd.server_close()
- self.join()
+ if self.is_ready.is_set():
+ if hasattr(self, "httpd"):
+ # Stop the WSGI server
+ self.httpd.shutdown()
+ self.httpd.server_close()
+ self.join()
}}}

My theory is that the "live server never became ready" situation I
simulated above is similar to the situation we're seeing on CI where a
database lock entails a failure to start a live server thread.
----
Then for one of the underlying assertion failures, I don't know how I feel
about masking a real problem, but we could probably reduce the chance of
failing jobs by adjusting `test_in_memory_database_lock()` to use the
`other` database instead of the `default`. It would still cover the code
it's testing, but it would just have a much smaller chance of interacting
poorly with other tests.
--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver mode
-------------------------------------+-------------------------------------

Reporter: Jacob Walls | Owner: Jacob
Type: | Walls

Cleanup/optimization | Status: assigned
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
parallel |

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* has_patch: 0 => 1
* owner: Kundan Yadav => Jacob Walls

Comment:

[https://github.com/django/django/pull/21029 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:5>

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver mode
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
parallel |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jacob Walls):

Replying to [comment:3 Jacob Walls]:

> we could probably reduce the chance of failing jobs by adjusting
test_in_memory_database_lock() to use the other database instead of the

default.

As Simon surmised on the above PR, this didn't help anything. (The PR
still suffered from occasional failures in this test after trying this
trick. I removed the speculative trick.)
--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:6>

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver mode
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
parallel |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jacob Walls <jacobtylerwalls@…>):

In [changeset:"9c9a43b45e07d2a2dcd3dd08681f7f5cecd82ab6" 9c9a43b4]:
{{{#!CommitTicketReference repository=""
revision="9c9a43b45e07d2a2dcd3dd08681f7f5cecd82ab6"
Refs #36770 -- Preferred addCleanup() in live server tests.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:8>

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver mode
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
parallel |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Jacob Walls <jacobtylerwalls@…>):

In [changeset:"afa026cd80a2388255a137a274568aef09f9fee7" afa026c]:
{{{#!CommitTicketReference repository=""
revision="afa026cd80a2388255a137a274568aef09f9fee7"
Refs #36770 -- Skipped test_in_memory_database_lock().

Skip pending some investigation.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:9>

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver mode
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
parallel |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Jacob Walls <jacobtylerwalls@…>):

In [changeset:"6c9ef6282e65a79daba4dc2146e8b825b4ddadb9" 6c9ef62]:
{{{#!CommitTicketReference repository=""
revision="6c9ef6282e65a79daba4dc2146e8b825b4ddadb9"
Refs #36770 -- Guarded against an endless wait in
LiveServerThread.terminate().

terminate() shouldn't assume the main server was started. (A deadlock
from mishandling of in-memory SQLite databases may have occurred.)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:7>

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver mode
-------------------------------------+-------------------------------------

Reporter: Jacob Walls | Owner: (none)
Type: | Status: new
Cleanup/optimization |

Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
parallel |

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* has_patch: 1 => 0
* owner: Jacob Walls => (none)
* status: assigned => new

--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:10>

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver mode
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner:

Type: | SnippyCodes
Cleanup/optimization | Status: assigned

Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
parallel |

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by SnippyCodes):

* has_patch: 0 => 1

* owner: (none) => SnippyCodes
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:11>

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver mode
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner:
Type: | SnippyCodes
Cleanup/optimization | Status: assigned
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
parallel |
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Simon Charette):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:12>

[Django]

#36770: SQLite threading tests are flaky when parallel test suite runs in
forkserver mode
-------------------------------------+-------------------------------------

Reporter: Jacob Walls | Owner: (none)
Type: | Status: new
Cleanup/optimization |

Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: 3.14, forkserver, | Triage Stage: Accepted
parallel |

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* has_patch: 1 => 0

* needs_better_patch: 1 => 0
* owner: SnippyCodes => (none)

* status: assigned => new

Comment:

Per PR comment, attention is being placed on #27734 instead.
--
Ticket URL: <https://code.djangoproject.com/ticket/36770#comment:13>