[Django] #34825: SQLite database files are not destroyed after tests

[Django]

#34825: SQLite database files are not destroyed after tests
---------------------------------------------+------------------------
Reporter: Jacob Walls | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: dev
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
After running the Django unit tests, I'm left with 8 .sqlite3 files like
so.

Bisected to 3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc.

{{{
tests % python3.11 runtests.py admin_inlines
Testing against Django installed in '/Users/jwalls/django/django' with up
to 8 processes
Found 82 test(s).
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
System check identified no issues (0 silenced).
.ssssssssssss.....................................................................
----------------------------------------------------------------------
Ran 82 tests in 1.419s

OK (skipped=12)
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
tests % git status
On branch main
Your branch is ahead of 'origin/main' by 86 commits.
(use "git push" to publish your local commits)

Untracked files:
(use "git add <file>..." to include in what will be committed)
other_1.sqlite3
other_2.sqlite3
other_3.sqlite3
other_4.sqlite3
other_5.sqlite3
other_6.sqlite3
other_7.sqlite3
other_8.sqlite3

nothing added to commit but untracked files present (use "git add" to
track)
}}}

----

Interestingly, I set a breakpoint at `_destroy_test_db()` and found this:
{{{
(Pdb) test_database_name
'file:memorydb_default?mode=memory&cache=shared'
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/34825>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#34825: SQLite database files are not destroyed after tests

-----------------------------------+--------------------------------------

Reporter: Jacob Walls | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: dev

Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------------+--------------------------------------

Comment (by David Sanders):

What's interesting is if you do `--parallel=1` then they aren't left
around 🤔

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:1>

[Django]

#34825: SQLite database files are not destroyed after tests
-----------------------------------+--------------------------------------
Reporter: Jacob Walls | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by David Sanders):

I'm new to the test framework but here's what I found when doing some
investigating:

Looks like the issue is with the new `setup_worker_connection()` method
for the sqlite3 backend in that commit you bisected.

It attempts to copy from `other_<worker-id>.sqlite3` into memory … but it
doesn't exist because the test runner doesn't create databases for the
"other" alias. The default behaviour of `sqlite3.connect()` is to create
a database because the default mode is `mode=rwc`, `c` being for create.

We can apparently change the mode to `mode=rw` in which case an exception
is raised. Or we can adjust the logic to use the same settings as "fork"
if the file doesn't exist. 🤔

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:2>

[Django]

#34825: SQLite database files are not destroyed after tests
-----------------------------------+------------------------------------

Reporter: Jacob Walls | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------

Changes (by David Sanders):

* has_patch: 0 => 1
* stage: Unreviewed => Accepted

Comment:

PR demonstrating my comment:
https://github.com/django/django/pull/17242/files

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:3>

[Django]

#34825: SQLite database files are not destroyed after tests

-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned

Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* owner: nobody => David Sanders
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:4>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia Bidart):

I'm not being able to reproduce, I'm using python3.11 with latest `main`
on a Manjaro system:

{{{
$ python3.11 runtests.py admin_inlines

Testing against Django installed in

'/home/nessita/fellowship/django/django' with up to 4 processes

Found 82 test(s).
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...

System check identified no issues (0 silenced).

..............ssssssssssss........................................................
----------------------------------------------------------------------
Ran 82 tests in 1.928s

OK (skipped=12)
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...

$ git st
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

$ ls *sqlite*
test_sqlite.py
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:5>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak):

> I'm not being able to reproduce, I'm using python3.11 with latest main
on a Manjaro system:

It's an issue with "spawn" method, so it's reproducible on Windows and
MacOS.

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:6>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by David Sanders):

I haven't tried this but you might also be able to set the start method in
runtests.py and change to spawn:
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:7>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia Bidart):

Thank you everyone, I was able to reproduce with this diff:

{{{#!diff
diff --git a/tests/runtests.py b/tests/runtests.py
index 2eb7490170..ad63e6a286 100755
--- a/tests/runtests.py
+++ b/tests/runtests.py
@@ -744,6 +744,7 @@ if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_sqlite")
options.settings = os.environ["DJANGO_SETTINGS_MODULE"]

+ multiprocessing.set_start_method("spawn")
if options.selenium:
if multiprocessing.get_start_method() == "spawn" and
options.parallel != 1:
parser.error(
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:8>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:9>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by faizan2700):

Please assign me this, I find it interesting.

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:10>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by David Sanders):

Hi faizan2700, unfortunately it's already assigned & fixed. Your best bet
is to search through tickets that are unassigned if you'd like to
contribute.

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:11>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* stage: Ready for checkin => Accepted

Comment:

Setting as Accepted until the conversation in the PR are resolved.

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:12>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:13>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak):

* needs_better_patch: 1 => 0

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:14>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders

Type: Bug | Status: closed

Component: Testing framework | Version: dev

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

* status: assigned => closed
* resolution: => fixed

Comment:

In [changeset:"a5905b164dbf52e59fa646af9c3d523c0804d86a" a5905b16]:
{{{
#!CommitTicketReference repository=""
revision="a5905b164dbf52e59fa646af9c3d523c0804d86a"
Fixed #34825 -- Avoided setting unused connections when initializing
parallel workers.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:15>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: closed
Component: Testing framework | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak <felisiak.mariusz@…>):

In [changeset:"2128a73713735fb794ca6565fd5d7792293f5cfa" 2128a737]:
{{{
#!CommitTicketReference repository=""
revision="2128a73713735fb794ca6565fd5d7792293f5cfa"
Refs #34825 -- Made SQLite backend open source database in readonly mode
when using spawn.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:16>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: closed
Component: Testing framework | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak <felisiak.mariusz@…>):

In [changeset:"bc6d71d4de8a3f6a430ca084419e079f2e3ecf31" bc6d71d4]:
{{{
#!CommitTicketReference repository=""
revision="bc6d71d4de8a3f6a430ca084419e079f2e3ecf31"
[5.0.x] Fixed #34825 -- Avoided setting unused connections when
initializing parallel workers.

Backport of a5905b164dbf52e59fa646af9c3d523c0804d86a from main
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:17>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: closed
Component: Testing framework | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by GitHub <noreply@…>):

In [changeset:"68d0159b6dfce07f144045d56639c52066e8b90e" 68d0159b]:
{{{
#!CommitTicketReference repository=""
revision="68d0159b6dfce07f144045d56639c52066e8b90e"
Fixed #34903, Refs #34825 -- Made workers initialization respect empty set
of used connections.

Thanks to David Smith for the investigation & patch.

Regression in 2128a73713735fb794ca6565fd5d7792293f5cfa.
Follow up to a5905b164dbf52e59fa646af9c3d523c0804d86a.

Co-authored-by: David Sanders <shang.xia...@gmail.com>
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:18>

[Django]

#34825: SQLite database files are not destroyed after tests
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: David
| Sanders
Type: Bug | Status: closed
Component: Testing framework | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak <felisiak.mariusz@…>):

In [changeset:"08aa336af46f30a29577a96588eeb3b440cd6dad" 08aa336a]:
{{{
#!CommitTicketReference repository=""
revision="08aa336af46f30a29577a96588eeb3b440cd6dad"
[5.0.x] Fixed #34903, Refs #34825 -- Made workers initialization respect

empty set of used connections.

Thanks to David Smith for the investigation & patch.

Regression in 2128a73713735fb794ca6565fd5d7792293f5cfa.
Follow up to a5905b164dbf52e59fa646af9c3d523c0804d86a.

Co-authored-by: David Sanders <shang.xia...@gmail.com>
Backport of 68d0159b6dfce07f144045d56639c52066e8b90e from main
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/34825#comment:19>