[Django] #33280: [bug] migration don't work with multiple databases (sometimes)
Django
-------------------------------------+-------------------------------------
Reporter: | Owner: nobody
razielvamp666 |
Type: Bug | Status: new
Component: Database | Version: 2.2
layer (models, ORM) |
Severity: Normal | Keywords: orm migration
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
My configuration is:
{{{
python 3.7.1
}}}
{{{
Django==2.2.24
django-axes==5.16.0
django-bootstrap-modal-forms==1.3.1
django-celery-results==1.0.4
django-cors-headers==3.1.1
django-database-view==0.3.0
django-debug-toolbar==1.11
django-dynamic-db-router==0.3.0
django-ipware==3.0.7
django-redis-cache==3.0.0
django-redis-sessions==0.6.2
django-ses==1.0.3
django-storages==1.12.3
django-taggit==0.24.0
djangorestframework==3.12.4
pytest-django==4.4.0
}}}
I using Django with multiple databases and default DB is not set for
security reasons (I prefer to get error if Django will fallback to
`default` connection, because it means that current tenant DB is not used
for some reasons). so `default` connection is empty (`{}`).
When I run `python manage.py migrate --database=some_db_name`
I've get error:
{{{
Applying app_name.0154_migration_name...Traceback (most recent call
last):
File "manage.py", line 41, in <module>
execute_from_command_line(sys.argv)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/core/management/__init__.py", line 381, in
execute_from_command_line
utility.execute()
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/core/management/base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/core/management/commands/migrate.py", line 234, in handle
fake_initial=fake_initial,
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake,
fake_initial=fake_initial)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/migrations/executor.py", line 147, in
_migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake,
fake_initial=fake_initial)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/migrations/executor.py", line 245, in apply_migration
state = migration.apply(state, schema_editor)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/migrations/migration.py", line 124, in apply
operation.database_forwards(self.app_label, schema_editor, old_state,
project_state)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/dbview/helpers.py", line 21, in database_forwards
self._create_standard_view(model, schema_editor)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/dbview/helpers.py", line 60, in _create_standard_view
qs = str(model.view())
File
"/user/code/genki_labo/orm_django/models/novel_coronavirus_vaccine.py",
line 136, in view
return str(qs.query)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/models/sql/query.py", line 265, in __str__
sql, params = self.sql_with_params()
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/models/sql/query.py", line 273, in sql_with_params
return self.get_compiler(DEFAULT_DB_ALIAS).as_sql()
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/models/sql/compiler.py", line 474, in as_sql
extra_select, order_by, group_by = self.pre_sql_setup()
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/models/sql/compiler.py", line 54, in pre_sql_setup
self.setup_query()
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/models/sql/compiler.py", line 45, in setup_query
self.select, self.klass_info, self.annotation_col_map =
self.get_select()
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/models/sql/compiler.py", line 254, in get_select
sql, params = self.compile(col, select_format=True)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/models/sql/compiler.py", line 405, in compile
sql, params = node.as_sql(self, self.connection)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/models/expressions.py", line 737, in as_sql
return "%s.%s" % (qn(self.alias), qn(self.target.column)), []
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/models/sql/compiler.py", line 396, in
quote_name_unless_alias
r = self.connection.ops.quote_name(name)
File "/user/.pyenv/versions/gl/lib/python3.7/site-
packages/django/db/backends/dummy/base.py", line 20, in complain
raise ImproperlyConfigured("settings.DATABASES is improperly
configured. "
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is
improperly configured. Please supply the ENGINE value. Check settings
documentation for more details.
}}}
You can see that problem is here:
{{{
return self.get_compiler(DEFAULT_DB_ALIAS).as_sql()
}}}
`DEFAULT_DB_ALIAS` is `default` and it hardcoded.
Also `DEFAULT_DB_ALIAS` is used couple of times inside
`django/db/models/sql/query.py` (please search by `DEFAULT_DB_ALIAS` in
file to see details)
It should use CURRENT_DB as other parts of Django
Unfortunately I didn't check how migration determines current DB when
`--database=...` is set so can't recommend exact solution, but I may see
couple hints:
1. DB may be get from `--database=...` directly (But not sure if that is a
correct solution).
2. DB may be get from
[`DatabaseRouter`](https://docs.djangoproject.com/en/2.2/topics/db/multi-
db/#database-routers) as other parts of Django (It is looking like correct
solution)
3. Here should be `using` option to be able set connection DB directly
(for example, all functions inside `django/db/transaction.py` has `using`
argument to be able set connection name directly)
Right now I using workaround and set `from django.conf import settings;
settings.DATABASES['default'] = current_db_taken_from_cmd` if command is
`migrate`.
Not all migrations fail. I don't know reason why it happens only with
specific migrations, but I guess that it because this migration doing
`CreateView`
Migration file is looking like:
{{{
# Generated by Django 2.2.10 on 2021-10-06 04:17
from dbview.helpers import CreateView
import common.enums.some_enum
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('app_name', '0153_previous_migration'),
]
operations = [
migrations.CreateModel(
name='SomeName',
fields=[
('id', models.AutoField(auto_created=True,
primary_key=True, serialize=False, verbose_name='ID')),
('create_datetime',
models.DateTimeField(auto_now_add=True, verbose_name='A')),
('user', models.CharField(default='admin', max_length=32,
verbose_name='B')),
('update_datetime', models.DateTimeField(auto_now=True,
verbose_name='C')),
...
],
options={
'db_table': 'table_name',
'unique_together': {('other_table', 'table_name')},
},
),
CreateView(
name='SomeNameView',
fields=[
('table_name_1',
models.OneToOneField(on_delete=django.db.models.deletion.PROTECT,
primary_key=True, serialize=False, to='orm_django.ApplyingInformation')),
('table_name_2', models.IntegerField(verbose_name='A')),
('table_name_3', models.IntegerField(verbose_name='B')),
('table_name_3', models.IntegerField(verbose_name='C')),
...
],
options={
'db_table': 'table_view',
},
),
migrations.AddIndex(
model_name='modelname',
index=models.Index(fields=['field_A', 'field_B', ...],
name='index_name'),
),
]
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33280>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: razielvamp666 | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution: invalid
Keywords: orm migration | Triage Stage:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => invalid
Comment:
Thanks for this report. However a lot of things happen outside of Django,
you have many 3rd-party packages, a custom `Operation` in migrations, etc.
The main issue is that you're using the `query` attribute to generate an
SQL which is not recommended, documented, or supported. Moreover, this
helper has issues with parameters (see #25705) and doesn't work when the
`default` database is empty (see #25947). I'm not marking this as a
duplicate of #25947 as fixing it will not make `.query` the recommended
way to generate an SQL. I would use a raw SQL in your case.
--
Ticket URL: <https://code.djangoproject.com/ticket/33280#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: razielvamp666 | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution: invalid
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> My configuration is:
>
> {{{
> python 3.7.1
> }}}
>
New description:
--
--
Ticket URL: <https://code.djangoproject.com/ticket/33280#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: razielvamp666 | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution: invalid
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
New description:
My configuration is:
{{{
python 3.7.1
}}}
I've get error:
File "/user/code/app_name/app_name/models/file_name.py", line 136, in
{{{
return self.get_compiler(DEFAULT_DB_ALIAS).as_sql()
}}}
class Migration(migrations.Migration):
--
--
Ticket URL: <https://code.djangoproject.com/ticket/33280#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: razielvamp666 | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution: invalid
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by razielvamp666):
Replying to [comment:1 Mariusz Felisiak]:
> Thanks for this report. However a lot of things happen outside of
Django, you have many 3rd-party packages, a custom `Operation` in
migrations, etc. The main issue is that you're using the `query` attribute
to generate an SQL which is not recommended, documented, or supported.
Moreover, this helper has issues with parameters (see #25705) and doesn't
work when the `default` database is empty (see #25947). I'm not marking
this as a duplicate of #25947 as fixing it will not make `.query` the
recommended way to generate an SQL. I would use a raw SQL in your case.
>
Thank you for answer.
Sorry, I didn't notice that was used little bit our code too. Not just
standard libraries.
But, Is I will wrong if I say:
1. Django support multiple databases
2. `django/db/models/sql/query.py` not support multiple databases because
it uses `DEFAULT_DB_ALIAS` and no matter how and when it run. Result will
be always same for non `default` DB
3. `django/db/models/sql/query.py` is a standard part of Django
4. standard part of Django not support multiple databases
5. => Django not support multiple databases (at least for all features)
Isn't this a kind of bug?
--
Ticket URL: <https://code.djangoproject.com/ticket/33280#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: razielvamp666 | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution: invalid
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by razielvamp666):
And thank you for point to #25947
Yes, the actual problem is the same.
6 years as accepted bug :)
--
Ticket URL: <https://code.djangoproject.com/ticket/33280#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: razielvamp666 | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution: invalid
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak):
> But, Is I will wrong if I say:
>
> 1. Django support multiple databases
> 2. `django/db/models/sql/query.py` not support multiple databases
because it uses `DEFAULT_DB_ALIAS` and no matter how and when it run.
Result will be always same for non `default` DB
> 3. `django/db/models/sql/query.py` is a standard part of Django
> 4. standard part of Django not support multiple databases
> 5. => Django not support multiple databases (at least for all features)
>
> Isn't this a kind of bug?
Using `DEFAULT_DB_ALIAS` is not always against supporting multiple
databases, sometimes is adequate. It's not always easy, feasible, or
necessary to pass a current connection. You're making too strong
assumptions. Also, as far as I'm concerned we don't claim that Django
support interacting with multiple databases everywhere, see
[https://docs.djangoproject.com/en/stable/topics/db/multi-db/#multiple-
databases docs].
--
Ticket URL: <https://code.djangoproject.com/ticket/33280#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: razielvamp666 | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution: invalid
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak):
Replying to [comment:5 razielvamp666]:
> 6 years as accepted bug :)
Feel-free to prepare a patch.
--
Ticket URL: <https://code.djangoproject.com/ticket/33280#comment:7>