[Django] #36496: SQLite test database path not recognised when running tests in parallel

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-------------------------------------+-------------------------------------
Reporter: Damian Posener | Type: Bug
Status: new | Component: Testing
| framework
Version: 5.2 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
How to replicate:

1. Have a DATABASES configuration with one or more SQLite DBs in
directories, like this:
{{{
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db' / 'default' / 'db.sqlite3',
},
'other': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db' / 'other' / 'db.sqlite3',
}
}
}}}
2. Run tests with both the `--parallel=auto` and `--keepdb` flags, i.e.
`python manage.py test --keepdb --parallel=auto`

Expected behaviour: `.sqlite3` files should be kept in `db/default` and
`db/other` directories

Actual behaviour: `default_X.sqlite3` and `other_X.sqlite3` files are
saved in the root directory instead

Why is this a problem? When running multiple test suites at once, this
leads to filename collision if two test suites are using the same database
name(s). It is impossible to run two Django test suites in parallel from
the same directory without encountering this issue.

Setting `DATABASES['<alias>']['TEST']['NAME']` does not seem to solve this
either.

I've put together an example project to demonstrate the issue. Simply run
`python manage.py test --keepdb --parallel=auto` from the base directory.

Happy to answer any other questions. :)
--
Ticket URL: <https://code.djangoproject.com/ticket/36496>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36496: SQLite test database path not recognised when running tests in parallel

-----------------------------------+--------------------------------------
Reporter: Damian Posener | Owner: (none)

Type: Bug | Status: new
Component: Testing framework | Version: 5.2

Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------------+--------------------------------------
Changes (by Damian Posener):

* Attachment "example.zip" added.

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+--------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Changes (by abhishek1999):

* cc: abhishek1999 (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:1>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+--------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by abhishek1999):

Hi 👋 — I’m trying to reproduce this bug to help confirm .

Here’s what I did so far:

1. Created a test project with the `DATABASES` config exactly as
described.
2. Ran `migrate` — the `.sqlite3` file was created correctly in
`db/default/`.
3. Added a simple `TestCase` with multiple dummy tests.
4. Ran `python manage.py test --keepdb --parallel=auto`.

**Environment:**
- Django version: `6.0.dev20250612075530` (editable install)
- Python version: `3.12.11`
- SQLite version: `3.43.2`

I only see `db/default/db.sqlite3` — no additional `default_1.sqlite3` or
`other_1.sqlite3` files appear in the project root.

Am I missing any step to trigger the parallel test DB creation?
Any suggestions are appreciated — thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:2>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+--------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Simon Charette):

> Why is this a problem? When running multiple test suites at once, this
leads to filename collision if two test suites are using the same database
name(s). It is impossible to run two Django test suites in parallel from
the same directory without encountering this issue.

I don't think that's something Django's test machinery claims to supports
by the way, I can't think of multiple things that would break running
multiple test suites against the same checkout of a projects.

> Am I missing any step to trigger the parallel test DB creation?

You need to have at least two `TestCase` otherwise
[https://github.com/django/django/blob/6df19412aabb7d969f5eab4b2ff41269de89b233/django/test/runner.py#L989-L991
these is nothing to parallelize].
--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:3>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+--------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Damian Posener):

Replying to [comment:2 abhishek1999]:

> Hi 👋 — I’m trying to reproduce this bug to help confirm .
>
> Here’s what I did so far:
>
> 1. Created a test project with the `DATABASES` config exactly as
described.
> 2. Ran `migrate` — the `.sqlite3` file was created correctly in
`db/default/`.
> 3. Added a simple `TestCase` with multiple dummy tests.
> 4. Ran `python manage.py test --keepdb --parallel=auto`.
>
> **Environment:**
> - Django version: `6.0.dev20250612075530` (editable install)
> - Python version: `3.12.11`
> - SQLite version: `3.43.2`
>
> I only see `db/default/db.sqlite3` — no additional `default_1.sqlite3`
or `other_1.sqlite3` files appear in the project root.
>

> Am I missing any step to trigger the parallel test DB creation?

> Any suggestions are appreciated — thanks!

There is a demo project attached to this ticket as a zip file. If you
extract that and run tests you can replicate the issue hopefully.
--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:4>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+--------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------
Comment (by Damian Posener):

Replying to [comment:3 Simon Charette]:

> I don't think that's something Django's test machinery claims to
supports by the way, I can't think of multiple things that would break
running multiple test suites against the same checkout of a projects.

We are using Django as a CMS for 14 different websites, and the nature of
that beast is a lot of separate apps, each with their own separate test
suites. Being able to run app tests in parallel saves us a bunch of time,
but this database thing is a bit of a thorn in our side.

I think there's also some (probably minor) security risks around Django
simply dumping it's databases copies in the current working directory. For
one the directory might not actually be writable. If it is writable, it
may contain other `.sqlite3` files that aren't meant to be overwritten
(Django does not seem to check whether the DBs are its own and will
happily overwrite existing ones).

Personally I'd be happy if there was at least a way to specify where
Django should store the files, but there seems to be no other option than
to use the current working dir.
--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:5>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+--------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Simon Charette):

> Personally I'd be happy if there was at least a way to specify where
Django should store the files, but there seems to be no other option than
to use the current working dir.

Have you tried using the explicit `TEST_NAME`
[https://docs.djangoproject.com/en/5.2/ref/settings/#std-setting-TEST_NAME
setting]?
--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:6>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+------------------------------------

Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------

Changes (by Simon Charette):

* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:7>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------

Comment (by abhishek1999):

> You need to have at least two `TestCase` otherwise
[https://github.com/django/django/blob/6df19412aabb7d969f5eab4b2ff41269de89b233/django/test/runner.py#L989-L991
these is nothing to parallelize].

Thanks, I was able to reproduce the issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:8>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Comment (by abhishek1999):

I tried this:

{{{
DATABASES = {
"default": {
"ENGINE": "django.db.backends.sqlite3",
"NAME": BASE_DIR / "db" / "default" / "db.sqlite3",

"TEST": {
"NAME": BASE_DIR / "db" / "default" / "test_db.sqlite3",

},
},
"other": {
"ENGINE": "django.db.backends.sqlite3",
"NAME": BASE_DIR / "db" / "other" / "db.sqlite3",

"TEST": {
"NAME": BASE_DIR / "db" / "other" / "test_db.sqlite3",
},
},
}
}}}

I added `['TEST']['NAME']` to keep test DB files in separate folders.
I ran:

{{{
python manage.py test --parallel=auto --keepdb
}}}
I got this error:

{{{
sqlite3.OperationalError: unable to open database file
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:9>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------

Comment (by Simon Charette):

Damian, do you have a requirement that parallel test databases are backed
by actual files instead of being in memory per worker process which should
ensure isolation?

What I mean is that if you want to avoid collisions until this gets fixed
you could simply set `DATABASES['<alias>']['TEST']['NAME'] = ':memory:'`
--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:10>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel
-----------------------------------+------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new
Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------

Comment (by Damian Posener):

Replying to [comment:10 Simon Charette]:

> Damian, do you have a requirement that parallel test databases are
backed by actual files instead of being in memory per worker process which
should ensure isolation?
>
> What I mean is that if you want to avoid collisions until this gets
fixed you could simply set `DATABASES['<alias>']['TEST']['NAME'] =

':memory:'` which is a common setup to speed up tests and ensure
isolation.

That would rather defeat the point of using `--keepdb`, don't you think?
:D We do run with in-memory DBs for a few applications, but that's an
inefficient choice when testing apps that have a load of migrations in
them.
--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:11>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel

-----------------------------------+--------------------------------------
Reporter: Damian Posener | Owner: Jason Hall
Type: Bug | Status: assigned

Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------------+--------------------------------------
Changes (by Jason Hall):

* owner: (none) => Jason Hall
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:12>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel

-----------------------------------+------------------------------------
Reporter: Damian Posener | Owner: (none)
Type: Bug | Status: new

Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------

Changes (by Jason Hall):

* owner: Jason Hall => (none)
* status: assigned => new

--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:13>

[Django]

#36496: SQLite test database path not recognised when running tests in parallel

-------------------------------------+-------------------------------------
Reporter: Damian Posener | Owner: Gangadhar
| Yadav
Type: Bug | Status: assigned

Component: Testing framework | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Gangadhar Yadav):

* owner: (none) => Gangadhar Yadav

* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36496#comment:14>