[Django] #26822: New migrations not applied on clone databases (sqlite) when using --parallel and --keepdb option
Django
--parallel and --keepdb option
-------------------------------+--------------------
Reporter: romgar | Owner: nobody
Type: Bug | Status: new
Component: Uncategorized | Version: 1.9
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------
I have noticed an issue when using `--parallel` and `--keepdb` option in
test environment.
The first time I run my tests with both options, everything is fine.
If I run again my tests after having added a new migration, this migration
is not applied to the clone databases, which makes the tests using these
cloned databases fail.
After looking a bit on `test.runner.setup_databases` function, I noticed
that the first database is created in `connection.creation.create_test_db`
(with correct migrations applied to it), and then the cloned database
creation happens in `connection.creation.clone_test_db`.
On sqlite, the function responsible of database cloning is
`db.backend.sqlite3.creation._clone_test_db`, and in case of `keepdb`
option, we don't touch the cloned databases:
{{{
if keepdb:
return
}}}
But then, the migrations applied to the default database are not present
in cloned ones.
I have noticed this behaviour on sqlite, I have not checked what happened
on other databases.
If we remove this `keepdb` condition, that will work but slow down the
test execution. A more interesting approach would probably be to apply
migrations to all databases, not just the default one.
Any thoughts about that? Or maybe I have missed some details?
--
Ticket URL: <https://code.djangoproject.com/ticket/26822>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
--parallel and --keepdb option
Reporter: romgar | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: => 0
* needs_docs: => 0
* component: Uncategorized => Testing framework
* needs_tests: => 0
* stage: Unreviewed => Accepted
Comment:
Seems like it would be helpful to fix that or document the limitation if
it's not feasible.
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:1>
Django
--parallel and --keepdb option
Reporter: romgar | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by romgar):
I will try to propose something then!
I have the feeling that applying `migrate` to all cloned databases seems
an interesting lead.
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:2>
Django
--parallel and --keepdb option
Reporter: romgar | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by romgar):
I have proposed a solution to fix this issue:
https://github.com/django/django/pull/6884
Tell me what you think about this cloned database deletion/creation, even
with `keepdb` option.
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:3>
Django
--parallel and --keepdb option
Reporter: romgar | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by romgar):
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:4>
Django
--parallel and --keepdb option
Reporter: romgar | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:5>
Django
--parallel and --keepdb option
Reporter: Romain Garrigues | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.9
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Simon Charette):
FWIW we worked around this issue by overriding our test suite runner's
`.parallel_test_suite.init_worker` to perform a `call_command('migrate',
verbosity=0)` after calling the Django defined `init_worker`. It feels
like it should be performed at the `create_test_db` level though.
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:6>
Django
--parallel and --keepdb option
Reporter: Romain Garrigues | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.9
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Sage Abdullah):
* cc: Sage Abdullah (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:7>
Django
--parallel and --keepdb option
Reporter: Romain Garrigues | Owner:
| Youngkwang Yang
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Youngkwang Yang):
* owner: nobody => Youngkwang Yang
* status: new => assigned
Comment:
I'd like to work on this issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:8>
Django
--parallel and --keepdb option
Reporter: Romain Garrigues | Owner:
| Youngkwang Yang
Type: Bug | Status: assigned
Component: Testing framework | Version: 1.9
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Youngkwang Yang):
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:9>
Django
--parallel and --keepdb option
Reporter: Romain Garrigues | Owner:
| Youngkwang Yang
Type: Bug | Status: assigned
Component: Testing framework | Version: 1.9
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Youngkwang Yang):
This issue still affects current versions. (I reproduced it again
recently.)
to reduce confusion, I’ve added documentation explaining the limitation.
-> https://github.com/django/django/pull/20187
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:10>
Django
--parallel and --keepdb option
Reporter: Romain Garrigues | Owner:
| Youngkwang Yang
Type: Bug | Status: assigned
Component: Testing framework | Version: 1.9
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Clément Escolano):
Also interested in this.
Right now, this is not possible to use `--parallel` and `--keepdb` in the
long run as sooner or later, some tests will break because they run on an
outdated clone.
Too bad the pull request https://github.com/django/django/pull/7551 was
abandoned as it was a simple patch that fixed the issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/26822#comment:11>