[Django] #26163: Wrong related fields in ._meta.get_fields with multiple database on foreignkey
Django
foreignkey
-------------------------------------+-------------------------------------
Reporter: aRkadeFR | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.9
(models, ORM) | Keywords: models, meta,
Severity: Normal | get_fields, fields, router,
| multiple, databases
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Hello,
I tried to find a similar ticket and didn't find it.
I setup a simple use case to show the error.
2 applications: appa and appb.
You override (by inheritance) django.contrib.auth.models.Group in <appa>,
you have a model ModelB in <appb> that has a foreign key to Group of
django.contrib.auth.
You setup a DatabaseRouter to route every <appa> model to the 'default'
database, and all <appb> model to the 'other' database.
If you ._meta.get_fields() on GroupA, you see the foreign key of the
ModelB which is related to django.contrib.auth in the other database. The
related field shouldn't be visible from the get_fields() on GroupA.
I got this error from debugging a "missing attribute" while Django was
emulating the DELETE ON_CASCADE in python and following the wrong parent
model to get then the related_fields.
Thanks for all the information and keep up the good work :)
{{{
# settings
DATABASE_ROUTERS = ['projecta.router.CustomRouter']
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST': 'localhost',
'PORT': '5432',
'NAME': 'default',
'USER': 'postgres_user',
'PASSWORD': 'pouetpouet',
},
'other': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST': 'localhost',
'PORT': '5432',
'NAME': 'other',
'USER': 'postgres_user',
'PASSWORD': 'pouetpouet',
}
}
# projecta/router.py
class CustomRouter(object):
""" CustomRouter """
def db_for_read(self, model, **hints):
if model._meta.app_label == 'appb':
return 'other'
if model._meta.app_label == 'auth':
return 'other'
return None
def db_for_write(self, model, **hints):
if model._meta.app_label == 'appb':
return 'other'
if model._meta.app_label == 'auth':
return 'other'
return None
def allow_relation(self, obj1, obj2, **hints):
if obj1._meta.app_label == 'auth' or \
obj2._meta.app_label == 'auth':
return True
return None
def allow_migrate(self, db, app_label, model=None, **hints):
""" auth need to go to the other database (for projectb) """
if app_label == 'appa':
return db == 'default'
if app_label == 'appb':
return db == 'other'
return None
# appa/models.py
from django.contrib.auth.models import Group
from django.db import models
class GroupA(Group):
extra_field = models.IntegerField()
# appb/models.py
from django.contrib.auth.models import Group
class ModelB(models.Model):
group = models.ForeignKey(Group, null=False)
# ipython
In [3]: GroupA._meta.get_fields()
Out[3]:
(<ManyToManyRel: auth.user>,
<ManyToOneRel: projectb.modelb>,
<...>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26163>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
foreignkey
-------------------------------------+-------------------------------------
Reporter: aRkadeFR | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.9
(models, ORM) |
Keywords: models, meta, | Triage Stage:
get_fields, fields, router, | Unreviewed
multiple, databases |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
Can you put together a sample project that reproduces the crash you
mentioned in the description?
I don't think `get_fields()` is or should be affected by database routers.
--
Ticket URL: <https://code.djangoproject.com/ticket/26163#comment:1>
Django
foreignkey
-------------------------------------+-------------------------------------
Reporter: aRkadeFR | Owner: nobody
Component: Database layer | Version: 1.9
(models, ORM) |
Keywords: models, meta, | Triage Stage:
get_fields, fields, router, | Unreviewed
multiple, databases |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => invalid
Comment:
Hi aRkadeFR,
As you might be aware the ORM implements concrete model inheritance using
[https://docs.djangoproject.com/en/1.9/topics/db/models/#multi-table-
inheritance multi-table inheritance] with a unique foreign key
(`OneToOneField`).
In your case a your `GroupA` model could also be explicitly defined as:
{{{#!python
from django.contrib.auth.models import Group
from django.db import models
class GroupA(Group):
group = models.OneToOneField(Group, parent_link=True)
}}}
Given your router definition it looks you're trying to define a cross-
database relation, [https://docs.djangoproject.com/en/1.9/topics/db/multi-
db/#cross-database-relations a feature documented as unsupported]. I can
see you even explicitly tried to work around this limitations by
overriding `allow_relation`.
Without a full traceback it's hard to figure out from where the failure
originates but I suppose a simple `GroupA.object.all()` should fail since
it will be resolved to an SQL query `JOIN`ing the `Group` table.
Now, the `_meta.get_field()` API '''always''' returned inherited fields
from parents so this doesn't look like a bug to me. If you want to make
sure the field is defined on a specific model you should access its
`model` attribute.
If you could provide a full traceback of the exception that was triggered
during the emulated cascade deletion you might be able to get help from
our support channels but given this is documented as unsupported I'll
close the ticket as invalid.
--
Ticket URL: <https://code.djangoproject.com/ticket/26163#comment:2>
Django
foreignkey
-------------------------------------+-------------------------------------
Reporter: aRkadeFR | Owner: nobody
Component: Database layer | Version: 1.9
(models, ORM) |
Keywords: models, meta, | Triage Stage:
get_fields, fields, router, | Unreviewed
multiple, databases |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by aRkadeFR):
We are not doing a cross database relation in our projects.
We have actually 2 distinct project, a "core" project, and a "side"
project we will call them.
The auth_group table is in both the "core" database, and the "side"
database (with extra fields in the "core" database).
The "side" project uses the "core" database as read-only models (through
the router).
And to setup the "side" project, we install both project (as app) at the
same time to makemigrations / migrate etc.
We have a missing attribute error or other error as provided in the
following stacktrace.
I think our real problem is to have 2 different "app" / "project" at the
same time with 2 different django.contrib.auth (in different database).
And we don't know how to set this up.
Can we have your thoughts? Thanks again for the answer
{{{
In [9]: GroupA.objects.all().delete()
---------------------------------------------------------------------------
ProgrammingError Traceback (most recent call
last)
<ipython-input-9-1976efc2649b> in <module>()
----> 1 GroupA.objects.all().delete()
/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in
delete(self)
598 collector = Collector(using=del_query.db)
599 collector.collect(del_query)
--> 600 deleted, _rows_count = collector.delete()
601
602 # Clear the result cache, in case this QuerySet gets
reused.
/usr/local/lib/python2.7/dist-packages/django/db/models/deletion.pyc in
delete(self)
290 # fast deletes
291 for qs in self.fast_deletes:
--> 292 count = qs._raw_delete(using=self.using)
293 deleted_counter[qs.model._meta.label] += count
294
/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in
_raw_delete(self, using)
612 query. No signals are sent, and there is no protection for
cascades.
613 """
--> 614 return sql.DeleteQuery(self.model).delete_qs(self, using)
615 _raw_delete.alters_data = True
616
/usr/local/lib/python2.7/dist-packages/django/db/models/sql/subqueries.pyc
in delete_qs(self, query, using)
79 self.where = self.where_class()
80 self.add_q(Q(pk__in=values))
---> 81 cursor = self.get_compiler(using).execute_sql(CURSOR)
82 return cursor.rowcount if cursor else 0
83
/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.pyc
in execute_sql(self, result_type)
846 cursor = self.connection.cursor()
847 try:
--> 848 cursor.execute(sql, params)
849 except Exception:
850 cursor.close()
/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.pyc in
execute(self, sql, params)
77 start = time()
78 try:
---> 79 return super(CursorDebugWrapper, self).execute(sql,
params)
80 finally:
81 stop = time()
/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.pyc in
execute(self, sql, params)
62 return self.cursor.execute(sql)
63 else:
---> 64 return self.cursor.execute(sql, params)
65
66 def executemany(self, sql, param_list):
/usr/local/lib/python2.7/dist-packages/django/db/utils.pyc in
__exit__(self, exc_type, exc_value, traceback)
93 if dj_exc_type not in (DataError, IntegrityError):
94 self.wrapper.errors_occurred = True
---> 95 six.reraise(dj_exc_type, dj_exc_value, traceback)
96
97 def __call__(self, func):
/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.pyc in
execute(self, sql, params)
62 return self.cursor.execute(sql)
63 else:
---> 64 return self.cursor.execute(sql, params)
65
66 def executemany(self, sql, param_list):
ProgrammingError: relation "appb_modelb" does not exist
LINE 1: DELETE FROM "appb_modelb" WHERE "appb_modelb"."group_id" IN ...
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26163#comment:3>
Django
foreignkey
-------------------------------------+-------------------------------------
Reporter: aRkadeFR | Owner: nobody
Component: Database layer | Version: 1.9
(models, ORM) |
Keywords: models, meta, | Triage Stage:
get_fields, fields, router, | Unreviewed
multiple, databases |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by charettes):
Hi aRkadeFR,
From what I can see `GroupA` deletion triggered cascade deletion attempts
of `ModelB` which is normal because of the FK relationships you defined
(`GroupA` < -- o2o --> `Group` <-- fk -- `ModelB`).
Because of your router definition the `app_modelb` table is not created on
the `default` database and thus defines a cross-database relationship
(even if you `auth_group` table is in both databases) with `GroupA` in the
context of the database used for its writes (`router.db_for_write(GroupA)
is None` -> `default`).
I won't go into more details about you question because this ticket
tracker is not a support channel but an application to track bugs. If you
want to get help to set things up please submit your question to the
mailing list or join the #django IRC channel.
--
Ticket URL: <https://code.djangoproject.com/ticket/26163#comment:4>