[Django] #23624: Regression in ManyToManyField with through, runtime-generated models
Django
----------------------------------------------+--------------------
Reporter: ludoo | Owner: nobody
Type: Bug | Status: new
Component: Database layer (models, ORM) | Version: 1.7
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+--------------------
I admit my situation is a corner case: an app interfacing with a legacy
db, where I need to extend an abstract model to contextualize it to
identical tables, named differently (eg obj_1 table, obj_2 table, etc.),
and to make things worse the number of tables is only known at runtime.
I am then dynamically generating a module for each set of models, and
while things work great in Django 1.6, in 1.7 ManyToManyField using a
through model breaks.
I am attaching a sample models.py file that illustrates the issue. I have
not yet found the time to delve into what's happening, I hope someone with
a better understanding of all the related stuff steps in as it's something
I have never looked into.
--
Ticket URL: <https://code.djangoproject.com/ticket/23624>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: ludoo | Owner: nobody
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by collinanderson):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I trace back would also help.
--
Ticket URL: <https://code.djangoproject.com/ticket/23624#comment:1>
Django
Reporter: ludoo | Owner: nobody
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Description changed by ludoo:
Old description:
> I admit my situation is a corner case: an app interfacing with a legacy
> db, where I need to extend an abstract model to contextualize it to
> identical tables, named differently (eg obj_1 table, obj_2 table, etc.),
> and to make things worse the number of tables is only known at runtime.
>
> I am then dynamically generating a module for each set of models, and
> while things work great in Django 1.6, in 1.7 ManyToManyField using a
> through model breaks.
>
> I am attaching a sample models.py file that illustrates the issue. I have
> not yet found the time to delve into what's happening, I hope someone
> with a better understanding of all the related stuff steps in as it's
> something I have never looked into.
New description:
I admit my situation is a corner case: an app interfacing with a legacy
db, where I need to extend an abstract model to contextualize it to
identical tables, named differently (eg obj_1 table, obj_2 table, etc.),
and to make things worse the number of tables is only known at runtime.
I am then dynamically generating a module for each set of models, and
while things work great in Django 1.6, in 1.7 ManyToManyField using a
through model breaks.
I am attaching a sample models.py file that illustrates the issue. I have
not yet found the time to delve into what's happening, I hope someone with
a better understanding of all the related stuff steps in as it's something
I have never looked into.
What happens is that the manager seems to ignore the through model, here
is the traceback (actual call and models in the attached file):
{{{
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/ludo/Desktop/dev/venv/spritz/local/lib/python2.7/site-
packages/django/db/models/manager.py", line 191, in all
return self.get_queryset()
File "/home/ludo/Desktop/dev/venv/spritz/local/lib/python2.7/site-
packages/django/db/models/fields/related.py", line 887, in get_queryset
return qs._next_is_sticky().filter(**self.core_filters)
File "/home/ludo/Desktop/dev/venv/spritz/local/lib/python2.7/site-
packages/django/db/models/query.py", line 691, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/home/ludo/Desktop/dev/venv/spritz/local/lib/python2.7/site-
packages/django/db/models/query.py", line 709, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/home/ludo/Desktop/dev/venv/spritz/local/lib/python2.7/site-
packages/django/db/models/sql/query.py", line 1287, in add_q
clause, require_inner = self._add_q(where_part, self.used_aliases)
File "/home/ludo/Desktop/dev/venv/spritz/local/lib/python2.7/site-
packages/django/db/models/sql/query.py", line 1314, in _add_q
current_negated=current_negated, connector=connector)
File "/home/ludo/Desktop/dev/venv/spritz/local/lib/python2.7/site-
packages/django/db/models/sql/query.py", line 1138, in build_filter
lookups, parts, reffed_aggregate = self.solve_lookup_type(arg)
File "/home/ludo/Desktop/dev/venv/spritz/local/lib/python2.7/site-
packages/django/db/models/sql/query.py", line 1076, in solve_lookup_type
_, field, _, lookup_parts = self.names_to_path(lookup_splitted,
self.get_meta())
File "/home/ludo/Desktop/dev/venv/spritz/local/lib/python2.7/site-
packages/django/db/models/sql/query.py", line 1383, in names_to_path
self.raise_field_error(opts, name)
File "/home/ludo/Desktop/dev/venv/spritz/local/lib/python2.7/site-
packages/django/db/models/sql/query.py", line 1389, in raise_field_error
"Choices are: %s" % (name, ", ".join(available)))
FieldError: Cannot resolve keyword u'a' into field. Choices are: a_s, id,
name
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/23624#comment:2>
Django
Reporter: ludoo | Owner: nobody
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by ludoo):
Right, I put the traceback in a comment in the attached file but did not
think of putting it here, fixed that now.
--
Ticket URL: <https://code.djangoproject.com/ticket/23624#comment:3>
Django
Reporter: ludoo | Owner: nobody
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by ludoo):
Progress.
I have found out that the exception disappears if the m2m field name is
the same as the fk field name in the through object: the keys in the
_name_map of the model holding the m2m field are populated using the
through model's field names. If the m2m field is named differently, the
field is missing from the name map and is not found at query time.
--
Ticket URL: <https://code.djangoproject.com/ticket/23624#comment:4>
Django
Reporter: ludoo | Owner: nobody
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by bmispelon):
I've bisected the error to this commit:
9f13c3328199d2fa70235cdc63bb06b1efc5b117
Note that using Python3, you get a slightly more useful error message:
{{{
Traceback (most recent call last):
File "./django/db/models/fields/related.py", line 876, in get_queryset
return
self.instance._prefetched_objects_cache[self.prefetch_cache_name]
AttributeError: 'A' object has no attribute '_prefetched_objects_cache'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "t.py", line 20, in <module>
a.b_s.all() # works in 1.6, raises FieldError in 1.7
File "./django/db/models/manager.py", line 191, in all
return self.get_queryset()
File "./django/db/models/fields/related.py", line 882, in get_queryset
return qs._next_is_sticky().filter(**self.core_filters)
File "./django/db/models/query.py", line 691, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "./django/db/models/query.py", line 709, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "./django/db/models/sql/query.py", line 1287, in add_q
clause, require_inner = self._add_q(where_part, self.used_aliases)
File "./django/db/models/sql/query.py", line 1314, in _add_q
current_negated=current_negated, connector=connector)
File "./django/db/models/sql/query.py", line 1138, in build_filter
lookups, parts, reffed_aggregate = self.solve_lookup_type(arg)
File "./django/db/models/sql/query.py", line 1076, in solve_lookup_type
_, field, _, lookup_parts = self.names_to_path(lookup_splitted,
self.get_meta())
File "./django/db/models/sql/query.py", line 1383, in names_to_path
self.raise_field_error(opts, name)
File "./django/db/models/sql/query.py", line 1389, in raise_field_error
"Choices are: %s" % (name, ", ".join(available)))
django.core.exceptions.FieldError: Cannot resolve keyword 'a' into field.
Choices are: a_s, id, name
}}}
(another interesting thing is that while bisecting, I encountered a
different error and even a RecursionError).
Still not sure if this is a bug in Django or if your use-case is just not
supported. It certainly looks weird.
--
Ticket URL: <https://code.djangoproject.com/ticket/23624#comment:5>
Django
Reporter: ludoo | Owner: nobody
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by aaugustin):
When you suspect a regression caused by the app-loading refactor, I
recommend running under -Wall or -Werror.
Django sends warnings in cases that will become errors to make the
transition slightly smoother. Unfortunately, since
PendingDeprecationWarnings are silent by default, this makes it a bit
difficult to pinpoint the root cause ofo some errors.
--
Ticket URL: <https://code.djangoproject.com/ticket/23624#comment:6>
Django
Reporter: ludoo | Owner: nobody
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by ludoo):
Replying to [comment:5 bmispelon]:
> I've bisected the error to this commit:
9f13c3328199d2fa70235cdc63bb06b1efc5b117
Fantastic. The commit points to the solution: a run-time generated models
module does not belong to any app, so its models are not used to resolve
complex relations.
Creating an AppConfig instance at runtime with the correct name, and a
module with a 'models' attribute pointing to the run-time generated models
module is not enough though, as one also needs to do some of the stuff
that gets done in the app registry's populate method when it's
initialized. It's a few lines of code but it feels weird and unclean, it
would be ideal if the apps registry had a way to register apps after it
has been initialized.
> Still not sure if this is a bug in Django or if your use-case is just
not supported. It certainly looks weird.
I admit the use-case is a bit specialized, but it's something that used to
work in 1.6, and it can be easily made to work in 1.7 with the steps
outlined above. The only problem I see is that finding *how* it works is
almost impossible for a regular person (it took your involvement to
pinpoint the correct commit, etc.). Having a way to register dynamically
generated apps/models, and somewhere in the docs that states how to do it
would be enough for the few people needing it, I guess.
BTW, my use-case is the Wordpress multi-blog database (each blog has its
own set of tables, etc), so it's probably a db design used in lots of
other legacy databases.
--
Ticket URL: <https://code.djangoproject.com/ticket/23624#comment:7>
Django
Reporter: ludoo | Owner: nobody
Type: New feature | Status: closed
Component: Documentation | Version: 1.7
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* component: Database layer (models, ORM) => Documentation
* type: Bug => New feature
* resolution: => wontfix
Comment:
I'm going to mark as "won't fix" as there is no official support for
runtime-generated models as far as I know.
--
Ticket URL: <https://code.djangoproject.com/ticket/23624#comment:8>