[Django] #29052: running test command will truncate the data source database at some condition
Django
condition
---------------------------------------------+------------------------
Reporter: Muse | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 2.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
= Reappear
When your database config look like following code, reader is the same as
default.
"default" is the data source database.
{{{
#!python
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'test',
'USER': 'root',
'PASSWORD': 'xxxx.',
'HOST': "127.0.0.1",
'PORT': '',
"TEST": {
"COLLATION": "utf8_general_ci",
"CHARSET": "utf8",
},
},
'reader': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'test',
'USER': 'root',
'PASSWORD': 'xxxx.',
'HOST': "127.0.0.1",
'PORT': '',
"TEST": {
"COLLATION": "utf8_general_ci",
"CHARSET": "utf8",
},
}
}
}}}
When I run ./manage.py test, this's a chance that db test will be
truncated, all data will be removed.
My project run under 1.11.2, but I tested on 2.0.1, problem remained.
= Trace
I following the code, when test database created, there's a method at
django/test/testcases.py was called by unittest.
{{{
#!python
def _fixture_teardown(self):
# Allow TRUNCATE ... CASCADE and don't emit the post_migrate
signal
# when flushing only a subset of the apps
for db_name in self._databases_names(include_mirrors=False):
# Flush the database
inhibit_post_migrate = (
self.available_apps is not None or
( # Inhibit the post_migrate signal when using
serialized
# rollback to avoid trying to recreate the serialized
data.
self.serialized_rollback and
hasattr(connections[db_name],
'_test_serialized_contents')
)
)
call_command('flush', verbosity=0, interactive=False,
database=db_name, reset_sequences=False,
allow_cascade=self.available_apps is not None,
inhibit_post_migrate=inhibit_post_migrate)
}}}
Calling command flush, there's my database murderer. When calling flush,
they using db test not using test_test.
I kept tracing, when testing framework start to initialize environment,
there's a function "setup_databases"(django/test/utils.py) called, it used
to create test database.
{{{
#!python
def setup_databases(verbosity, interactive, keepdb=False, debug_sql=False,
parallel=0, **kwargs):
"""
Create the test databases.
"""
test_databases, mirrored_aliases = get_unique_databases_and_mirrors()
old_names = []
for signature, (db_name, aliases) in test_databases.items():
first_alias = None
for alias in aliases:
connection = connections[alias]
old_names.append((connection, db_name, first_alias is None))
# Actually create the database for the first connection
if first_alias is None:
first_alias = alias
connection.creation.create_test_db(
verbosity=verbosity,
autoclobber=not interactive,
keepdb=keepdb,
serialize=connection.settings_dict.get('TEST',
{}).get('SERIALIZE', True),
)
if parallel > 1:
for index in range(parallel):
connection.creation.clone_test_db(
number=index + 1,
verbosity=verbosity,
keepdb=keepdb,
)
# Configure all other connections as mirrors of the first one
else:
connections[alias].creation.set_as_test_mirror(connections[first_alias].settings_dict)
}}}
At my case, aliases looks like {"reader", "default"} or {"default",
"reader"}.
When default was the first one, create_test_db made
connections.get("default") point to test_test.
On the contrary, connections.get("reader") point to test_test. ** But **,
connections.get("default") still point to test.
When choosing which database to flush, if just one database. Just using
default.
{{{
#!python
@classmethod
def _databases_names(cls, include_mirrors=True):
# If the test case has a multi_db=True flag, act on all databases,
# including mirrors or not. Otherwise, just on the default DB.
if getattr(cls, 'multi_db', False):
return [
alias for alias in connections
if include_mirrors or not
connections[alias].settings_dict['TEST']['MIRROR']
]
else:
return [DEFAULT_DB_ALIAS]
}}}
But default still point to test, not test_test, so, all data gone.
--
Ticket URL: <https://code.djangoproject.com/ticket/29052>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
point to the same database
-----------------------------------+------------------------------------
Reporter: Muse | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 2.0
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* stage: Unreviewed => Accepted
Comment:
I haven't reproduced this, but the report makes sense.
--
Ticket URL: <https://code.djangoproject.com/ticket/29052#comment:1>
Django
point to the same database
-----------------------------------+------------------------------------
Type: Bug | Status: new
Component: Testing framework | Version: 2.0
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Urth):
I've created a pull request which fixes this ticket by fully creating the
test databases and patching the settings of all aliases.
https://github.com/django/django/pull/13507
--
Ticket URL: <https://code.djangoproject.com/ticket/29052#comment:2>
Django
point to the same database
-----------------------------------+------------------------------------
Type: Bug | Status: assigned
Component: Testing framework | Version: 2.0
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* owner: nobody => Urth
* needs_better_patch: 0 => 1
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/29052#comment:4>
Django
point to the same database
Reporter: Muse | Owner: Urth
Type: Bug | Status: assigned
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/29052#comment:5>
Django
point to the same database
-------------------------------------+-------------------------------------
Reporter: Muse | Owner: Urth
Component: Testing framework | Version: 2.0
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"06e5f7ae1639f1e275e7cc1076dc70ca3ebaa946" 06e5f7ae]:
{{{
#!CommitTicketReference repository=""
revision="06e5f7ae1639f1e275e7cc1076dc70ca3ebaa946"
Fixed #29052 -- Made test database creation preserve alias order and
prefer the "default" database.
This fixes flushing test databases when two aliases point to the same
database.
Use a list() to store the test database aliases so the order remains
stable by following the order of the connections. Also, always use the
"default" database alias as the first alias to accommodate `migrate`.
Previously `migrate` could be executed on a secondary alias which
caused truncating the "default" database.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29052#comment:6>
Django
point to the same database
-------------------------------------+-------------------------------------
Reporter: Muse | Owner: Urth
Type: Bug | Status: closed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"b89ce413f2a95f22788efb01d441f7ab3b506003" b89ce413]:
{{{
#!CommitTicketReference repository=""
revision="b89ce413f2a95f22788efb01d441f7ab3b506003"
[3.2.x] Fixed #29052 -- Made test database creation preserve alias order
and prefer the "default" database.
This fixes flushing test databases when two aliases point to the same
database.
Use a list() to store the test database aliases so the order remains
stable by following the order of the connections. Also, always use the
"default" database alias as the first alias to accommodate `migrate`.
Previously `migrate` could be executed on a secondary alias which
caused truncating the "default" database.
Backport of 06e5f7ae1639f1e275e7cc1076dc70ca3ebaa946 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29052#comment:7>