[Django] #25747: djorm-ext-filtered-contenttypes impossible on Django 1.9
Django
-------------------------------------+-------------------------------------
Reporter: mpasternak | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.9b1
(models, ORM) | Keywords: postgresql db
Severity: Normal | contenttypes
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Hi,
I'm the author of a small package, that allows you to filter
GenericForeignKey fields using custom query magic introduced in Django
1.7, https://github.com/mpasternak/djorm-ext-filtered-contenttypes/ . It
works on Django 1.7 and 1.8.
On 1.9 my package is impossible to implement. Reason:
https://github.com/django/django/commit/9ed82154bd0bd01c6195942db84302e791ad366f
which fixed https://code.djangoproject.com/ticket/23791 .
I realize, that the fix for #23791 is very important and I'm glad this bug
was solved. I have nothing against it and it's cool that it was
implemented; having OneToOne as a primary key seems an interesting use
case.
On the other hand, I'd love to be able to use my package on Django 1.9
without too much hackery (there's already some involved).
The reason for GenericForeignKey filtering and for that app of mine... in
PostgreSQL you can create a compound index of (content_type_id, object_id)
and use it for querying. If I can write Python code like:
{{{
StorageRecord.objects.filter(object__in=OtherModel.objects.filter(...))
}}}
... and my package can translate it to:
{{{
SELECT * FROM app_storage_record WHERE (content_type_id, object_id) IN
(SELECT content_type_id, object_id FROM app_other_model WHERE ... )
}}}
... then PostgreSQL will be able to use compound index on
(content_type_id, object_id) and the query will be insanely fast.
I use this to create a large cache table for my app, that holds records
from 5 other tables. It uses GenericForeignKey to hold references to those
tables. I use my FilteredGenericForeignKey field to filter stuff in that
big table. On PostgreSQL with reasonable indexes it is very fast.
If you happen to download my project, just please run tox and see the
problem. 1.7 and 1.8 work fine. On 1.9 I get bugs like:
{{{
======================================================================
ERROR: test_single_object
(filtered_contenttypes.tests.test_fields.TestFilteredContentTypes)
----------------------------------------------------------------------
Traceback (most recent call last):
File "../filtered_contenttypes/tests/test_fields.py", line 32, in
test_single_object
qry = StorageRecord.objects.filter(item=self.l)
File "/Users/mpasternak/Programowanie/djorm-ext-filtered-
contenttypes/.tox/py35-django19/lib/python3.5/site-
packages/django/db/models/manager.py", line 122, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/mpasternak/Programowanie/djorm-ext-filtered-
contenttypes/.tox/py35-django19/lib/python3.5/site-
packages/django/db/models/query.py", line 790, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/Users/mpasternak/Programowanie/djorm-ext-filtered-
contenttypes/.tox/py35-django19/lib/python3.5/site-
packages/django/db/models/query.py", line 808, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/Users/mpasternak/Programowanie/djorm-ext-filtered-
contenttypes/.tox/py35-django19/lib/python3.5/site-
packages/django/db/models/sql/query.py", line 1240, in add_q
clause, _ = self._add_q(q_object, self.used_aliases)
File "/Users/mpasternak/Programowanie/djorm-ext-filtered-
contenttypes/.tox/py35-django19/lib/python3.5/site-
packages/django/db/models/sql/query.py", line 1266, in _add_q
allow_joins=allow_joins, split_subq=split_subq,
File "/Users/mpasternak/Programowanie/djorm-ext-filtered-
contenttypes/.tox/py35-django19/lib/python3.5/site-
packages/django/db/models/sql/query.py", line 1174, in build_filter
self.check_related_objects(field, value, opts)
File "/Users/mpasternak/Programowanie/djorm-ext-filtered-
contenttypes/.tox/py35-django19/lib/python3.5/site-
packages/django/db/models/sql/query.py", line 1071, in
check_related_objects
self.check_query_object_type(value, opts, field)
File "/Users/mpasternak/Programowanie/djorm-ext-filtered-
contenttypes/.tox/py35-django19/lib/python3.5/site-
packages/django/db/models/sql/query.py", line 1055, in
check_query_object_type
(value, opts.object_name))
ValueError: Cannot query "Laptop object": Must be "StorageRecord"
instance.
----------------------------------------------------------------------
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25747>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: mpasternak | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.9b1
(models, ORM) |
Keywords: postgresql db | Triage Stage:
contenttypes | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
... on the other hand, I can just set is_relation to False on the field
and have it done.
On the other hand, I'd really LOVE someone from core team have a look at
my small project. Being able to filter GenericForeignKeys out of the box
is not only possible AND efficient (with PostgreSQL and some indexing), it
also gives a lot of extra power to the ORM layer.
--
Ticket URL: <https://code.djangoproject.com/ticket/25747#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: mpasternak | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.9b1
(models, ORM) |
Keywords: postgresql db | Triage Stage:
contenttypes | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by timgraham):
Is the requested functionality described in #3006?
--
Ticket URL: <https://code.djangoproject.com/ticket/25747#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: mpasternak | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.9b1
(models, ORM) |
Keywords: postgresql db | Triage Stage:
contenttypes | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by mpasternak):
#3006 is similar but different.
My app is a drop-in replacement for GenericForeignKey, you just replace
this with FilteredGenericForeignKey in model definition and you're set.
You don't need to use any functions, just the ORM. #3006 uses additional
function to achieve that.
Also, I'd need to have a look at queries generated by #3006. My module is
built for PostgreSQL and efficiency; if you index both (content_type_id,
object_id) fields, my module is going to produce queries that will make
PostgreSQL use that index.
Also, I don't support aggregate() and annotate() in my module.
--
Ticket URL: <https://code.djangoproject.com/ticket/25747#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: mpasternak | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.9b1
(models, ORM) |
Keywords: postgresql db | Triage Stage:
contenttypes | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by timgraham):
Do you propose to try to incorporate the functionality your package
provides in Django in some way. If so, I guess a new ticket for that
functionality would be clearer. I'm unsure if you want to keep this ticket
open at this point?
--
Ticket URL: <https://code.djangoproject.com/ticket/25747#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: mpasternak | Owner: nobody
Component: Database layer | Version: 1.9b1
(models, ORM) |
Keywords: postgresql db | Triage Stage:
contenttypes | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => needsinfo
--
Ticket URL: <https://code.djangoproject.com/ticket/25747#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: mpasternak | Owner: nobody
Component: Database layer | Version: 1.9b1
(models, ORM) |
Keywords: postgresql db | Triage Stage:
contenttypes | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by mpasternak):
I had a look at #3006. My package allows you to generate queries like:
SELECT ... WHERE (content_type_id, object_id) = (SELECT (1, id) FROM ...)
... but it does NOT allow you to filter stuff in ORM like the other
package does.
So, ATM I'd not incorporate this into Django. Both #3006 and djorm-ext-
filtered-contenttypes have their place. Merging those packages in some
form of a cool API and adding some docs would make a great addition to
Django. ATM I'm not going to work on that, perhaps in the future.
I think this is very good we had this short discussion right here; this
functionality is really useful in some corner cases and I guess there
won't be many people needing it.
--
Ticket URL: <https://code.djangoproject.com/ticket/25747#comment:6>