[Django] #31169: Parallel test mode not working on MacOS with recent versions of Python 3
Django
---------------------------------------------+------------------------
Reporter: Brandon Navra | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
Python 3.8 on MacOS has changed the default start method for the
multiprocessing module from `fork` to `spawn`:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-
methods.
When running tests with the `--parallel` flag, this causes the worker
processes to fail with {{{django.core.exceptions.AppRegistryNotReady: Apps
aren't loaded yet.}}} as they no longer have a copy of the parent memory
state. It can also cause the workers to fail to find the cloned dbs
({{django.db.utils.OperationalError: FATAL: database "xxx_1" does not
exist}}) as the db test prefix is missing.
I have attached a patch which changes `django.test.runner._init_worker`
(the worker initialiser for `ParallelTestSuite`) to run django.setup() and
set the db name to one with the `test_` prefix.
--
Ticket URL: <https://code.djangoproject.com/ticket/31169>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Brandon Navra | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Brandon Navra):
* Attachment
"Ensure_Django_is_setup_correctly_in_parallel_test_workers.patch" added.
Django
Reporter: Brandon Navra | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Brandon Navra:
Old description:
> Python 3.8 on MacOS has changed the default start method for the
> multiprocessing module from `fork` to `spawn`:
> https://docs.python.org/3/library/multiprocessing.html#contexts-and-
> start-methods.
>
> When running tests with the `--parallel` flag, this causes the worker
> processes to fail with {{{django.core.exceptions.AppRegistryNotReady:
> Apps aren't loaded yet.}}} as they no longer have a copy of the parent
> memory state. It can also cause the workers to fail to find the cloned
> dbs ({{django.db.utils.OperationalError: FATAL: database "xxx_1" does
> not exist}}) as the db test prefix is missing.
>
> I have attached a patch which changes `django.test.runner._init_worker`
> (the worker initialiser for `ParallelTestSuite`) to run django.setup()
> and set the db name to one with the `test_` prefix.
New description:
Python 3.8 on MacOS has changed the default start method for the
multiprocessing module from `fork` to `spawn`:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-
methods.
When running tests with the `--parallel` flag, this causes the worker
processes to fail with {{{django.core.exceptions.AppRegistryNotReady: Apps
aren't loaded yet.}}} as they no longer have a copy of the parent memory
state. It can also cause the workers to fail to find the cloned dbs (
{{django.db.utils.OperationalError: FATAL: database "xxx_1" does not
exist}} ) as the db test prefix is missing.
I have attached a patch which changes `django.test.runner._init_worker`
(the worker initialiser for `ParallelTestSuite`) to run django.setup() and
set the db name to one with the `test_` prefix.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:1>
Django
Reporter: Brandon Navra | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by felixxm):
`spawn()` is also a default method on Windows, and we don't encounter any
issues with it 🤔.
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:2>
Django
Reporter: Brandon Navra | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Brandon Navra):
I'm still trying to research the exact root cause. The Python issue which
triggered this change has snippets of info:
https://code.djangoproject.com/ticket/31169
but nothing conclusive. My theory is that the memory copying semantics
between MacOS and Windows are different and hence the spawn method doesn't
have identical behaviour between the two.
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:3>
Django
Reporter: Brandon Navra | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by felixxm):
Ahhh, sorry we don't use `parallel` on Windows.
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:4>
Django
method.
-----------------------------------+------------------------------------
Reporter: Brandon Navra | Owner: nobody
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* type: Bug => New feature
* version: 3.0 => master
* stage: Unreviewed => Accepted
Comment:
Parallel running is disabled on Windows:
{{{
def default_test_processes():
"""Default number of test processes when using the --parallel
option."""
# The current implementation of the parallel test runner requires
# multiprocessing to start subprocesses with fork().
if multiprocessing.get_start_method() != 'fork':
return 1
try:
return int(os.environ['DJANGO_TEST_PROCESSES'])
except KeyError:
return multiprocessing.cpu_count()
}}}
I'll accept this as a new feature: the limitation has been there since it
was implemented.
Brandon, your patch is tiny. Is it really that simple? We'd need tests and
a few other adjustments (like to the function above) but, fancy opening a
PR?
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:5>
Django
method.
-----------------------------------+------------------------------------
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson):
So this occurs on macOS 10.15. (I have 10.14 currently so can't experiment
there.)
Applying the patch on Windows, alas, doesn't immediately solve the issue,
but it is INSTALLED_APPS/AppRegistry errors that are raised, so it's going
to be in the right ball-park.
More investigating needed, but this would be a good one to land.
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:6>
Django
method.
-----------------------------------+------------------------------------
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Brandon Navra):
I created a PR with the changes from my patch:
https://github.com/django/django/pull/12321
FYI" I am on macOS 10.14.6
I'm not sure how best to adjust `default_test_processes` or what you'd
like tested
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:7>
Django
method.
-----------------------------------+------------------------------------
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson):
> FYI" I am on macOS 10.14.6
Super. I had a 3.7. env active. I can reproduce with Python 3.8.
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:8>
Django
method.
-----------------------------------+------------------------------------
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Brandon Navra):
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:9>
Django
method.
-----------------------------------+------------------------------------
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Thierry Bastian):
Thanks for the report. I had the same issue but did not find the root
cause in https://code.djangoproject.com/ticket/31116.
I would love to see that being resolved.
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:10>
Django
method.
-----------------------------------+------------------------------------
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:11>
Django
method.
-----------------------------------+------------------------------------
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by Peter Inglesby):
I ran into this while running the Django test suite, and when applying the
patch in [https://github.com/django/django/pull/12321 PR 12321], I get the
same problem with a different exception:
{{{
Traceback (most recent call last):
File
"/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py",
line 313, in _bootstrap
self.run()
File
"/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py",
line 108, in run
self._target(*self._args, **self._kwargs)
File
"/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/pool.py",
line 114, in worker
task = get()
File
"/Users/inglesp/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/queues.py",
line 358, in get
return _ForkingPickler.loads(res)
File "/Users/inglesp/src/django/django/tests/fixtures_regress/tests.py",
line 18, in <module>
from .models import (
File
"/Users/inglesp/src/django/django/tests/fixtures_regress/models.py", line
1, in <module>
from django.contrib.auth.models import User
File "/Users/inglesp/src/django/django/django/contrib/auth/models.py",
line 3, in <module>
from django.contrib.contenttypes.models import ContentType
File
"/Users/inglesp/src/django/django/django/contrib/contenttypes/models.py",
line 133, in <module>
class ContentType(models.Model):
File "/Users/inglesp/src/django/django/django/db/models/base.py", line
113, in __new__
raise RuntimeError(
RuntimeError: Model class django.contrib.contenttypes.models.ContentType
doesn't declare an explicit app_label and isn't in an application in
INSTALLED_APPS.
}}}
I'm on OSX 10.15 with Python 3.8.
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:12>
Django
method.
-----------------------------------+------------------------------------
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Abhijeet Viswa):
* cc: Abhijeet Viswa (added)
Comment:
An attempt at a fix: https://github.com/django/django/pull/12547
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:13>
Django
method.
-----------------------------------+------------------------------------
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Ichlasul Affan):
* cc: Ichlasul Affan (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:14>
Django
method.
Reporter: Brandon Navra | Owner: Ahmad A.
| Hussein
Type: New feature | Status: assigned
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Ahmad A. Hussein):
* owner: nobody => Ahmad A. Hussein
* needs_better_patch: 1 => 0
* has_patch: 1 => 0
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:15>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: Ahmad A.
| Hussein
Type: New feature | Status: assigned
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam (Chainz) Johnson):
For those looking for a workaround, here's how to add the appropriate call
to reset back to fork mode: https://adamj.eu/tech/2020/07/21/how-to-use-
djangos-parallel-testing-on-macos-with-python-3.8-plus/
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:16>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: Ahmad A.
| Hussein
Type: New feature | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Ahmad A. Hussein):
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:17>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: Ahmad A.
| Hussein
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Abhijeet Viswa):
* cc: Abhijeet Viswa (removed)
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:18>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: Ahmad A.
| Hussein
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Ryan Siemens):
* cc: Ryan Siemens (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:19>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: Ahmad A.
| Hussein
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* needs_better_patch: 0 => 1
Comment:
PR is working nicely on macOS but needs a rebase, and a refactoring for
review comments.
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:20>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: Ahmad A.
| Hussein
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Adam Wróbel):
* cc: Adam Wróbel (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:21>
Django
method.
Reporter: Brandon Navra | Owner: David Smith
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by David Smith):
* owner: Ahmad A. Hussein => David Smith
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:22>
Django
method.
-----------------------------------+---------------------------------------
Reporter: Brandon Navra | Owner: David Smith
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* needs_better_patch: 0 => 1
Comment:
A few issues on the PR to resolve, but looking very promising: 39.813s vs
249.146s on macOS, which is a bit of a speed-up 🙂
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:23>
Django
method.
-----------------------------------+---------------------------------------
Reporter: Brandon Navra | Owner: David Smith
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"ae91ecf6a1037fb67d14841b66ac19d4c2ccc4ac" ae91ecf6]:
{{{
#!CommitTicketReference repository=""
revision="ae91ecf6a1037fb67d14841b66ac19d4c2ccc4ac"
Refs #31169 -- Added DatabaseCreation.setup_worker_connection() hook.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:24>
Django
method.
-----------------------------------+---------------------------------------
Reporter: Brandon Navra | Owner: David Smith
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------------+---------------------------------------
* needs_better_patch: 0 => 1
* stage: Ready for checkin => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:27>
Django
method.
Reporter: Brandon Navra | Owner: David
| Smith
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by GitHub <noreply@…>):
In [changeset:"795da6306a048b18c0158496b0d49e8e4f197a32" 795da630]:
{{{
#!CommitTicketReference repository=""
revision="795da6306a048b18c0158496b0d49e8e4f197a32"
Refs #31169 -- Prevented infinite loop in tests on failures.
Regression in ae91ecf6a1037fb67d14841b66ac19d4c2ccc4ac.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:26>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: David
| Smith
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:25>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: David
| Smith
Type: New feature | Status: assigned
Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak):
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:28>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: David
| Smith
Component: Testing framework | Version: dev
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc" 3b3f38b]:
{{{
#!CommitTicketReference repository=""
revision="3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc"
Fixed #31169 -- Adapted the parallel test runner to use spawn.
Co-authored-by: Valz <ahmadah...@gmail.com>
Co-authored-by: Nick Pope <ni...@nickpope.me.uk>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:29>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: David
| Smith
Type: New feature | Status: closed
Component: Testing framework | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"ba298a32b30eb270ea0bf4f8fc208223d0b40bcd" ba298a32]:
{{{
#!CommitTicketReference repository=""
revision="ba298a32b30eb270ea0bf4f8fc208223d0b40bcd"
Refs #31169 -- Prevented infinite loop in parallel tests with custom test
runner when using spawn.
Regression in 3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc.
Co-Authored-By: Mariusz Felisiak <felisiak...@gmail.com>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:30>
Django
method.
-------------------------------------+-------------------------------------
Reporter: Brandon Navra | Owner: David
| Smith
Type: New feature | Status: closed
Component: Testing framework | Version: dev
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by GitHub <noreply@…>):
In [changeset:"20d575b101608c1475bc2881f83b04da0e68ef7d" 20d575b]:
{{{
#!CommitTicketReference repository=""
revision="20d575b101608c1475bc2881f83b04da0e68ef7d"
Refs #31169 -- Skipped test_get_test_db_clone_settings_not_supported on
not in-memory SQLite database.
multiprocessing's start method is checked only for in-memory SQLite
databases.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31169#comment:31>