Ticket 28087: Allow filtering and ordering with RawQuerySet

299 views
Skip to first unread message

Dmitriy Sintsov

unread,
Apr 20, 2017, 4:06:46 PM4/20/17
to Django developers (Contributions to Django itself)
Hi!
I implemented FilteredRawQuerySet which supports filter() and order_by() like model manager QuerySet.


Internally it contains RawQuerySet and QuerySet for the target model.

SELECT part of the query is borrowed from RawQuerySet, while WHERE and ORDER BY statements are generated via QuerySet query.

This feature is useful when the code uses programmatically generated arguments of filter() / order_by(), for example in class-based ListView with custom filtering and / or sorting.

Is such feature worth to be included to Django core?

I could just continue to use my own implementation (probably imperfect) however I am worrying about possible query generation code changes that could make my module incompatible in the future.

In case such queryset would be included to Django code, there are much more chances that the code which uses it would not break when updating to newer Django version.

Dmitriy

Adam Johnson

unread,
Apr 20, 2017, 6:02:40 PM4/20/17
to django-d...@googlegroups.com
I'm worried this doesn't work in the general case, as there are a lot of different query formats one can write with raw SQL, such as UNION, SQL variables, or other database specific formats. You have a lot of code there, but from what I can tell the code is simply setting up the RawSqlCompiler which adds WHERE etc. after the raw SQL, ignoring the fact that there could already be a WHERE in it - am I right?

Also - what do you find raw queries being needed for? In general the ORM is very capable these days and the general development focus on it is to make it cover even more of SQL's capabilities.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a20e043e-aec8-4806-9e6b-c5c669361c44%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Adam

Dmitriy Sintsov

unread,
Apr 21, 2017, 4:47:16 AM4/21/17
to Django developers (Contributions to Django itself)
I know that it does not work in the general case, but still having orthogonal processing of RAW queries with the same filter() / order_by() calls matches Django's DRY philosophy.
Raw queries which has hardcoded WHERE and ORDER BY statements will simply raise an error when another WHERE / ORDER BY statement is added by filter() / order_by().
But there is no point to specify WHERE manually when there is filter() available and such error brings no direct harm.
I implemented it mostly to use in paginated ListView-like CBVs with LEFT JOIN. Here is the example of real query from private project:

'SELECT isp_finances_profilerequisites.*, '
'isp_user_profile.user_id, isp_user_profile.last_name, isp_user_profile.first_name, isp_user_profile.is_verified, isp_user_profile.inn '
'FROM isp_user_profile '
'LEFT JOIN isp_finances_profilerequisites '
'ON isp_finances_profilerequisites.profile_id = isp_user_profile.user_id '
'JOIN auth_user '
'ON auth_user.id = isp_user_profile.user_id '

Is such RAW query (with working pagination and sorting) possible to convert to model manager QuerySet query?

On Friday, April 21, 2017 at 1:02:40 AM UTC+3, Adam Johnson wrote:
I'm worried this doesn't work in the general case, as there are a lot of different query formats one can write with raw SQL, such as UNION, SQL variables, or other database specific formats. You have a lot of code there, but from what I can tell the code is simply setting up the RawSqlCompiler which adds WHERE etc. after the raw SQL, ignoring the fact that there could already be a WHERE in it - am I right?

Also - what do you find raw queries being needed for? In general the ORM is very capable these days and the general development focus on it is to make it cover even more of SQL's capabilities.
On 20 April 2017 at 20:36, Dmitriy Sintsov <quest...@gmail.com> wrote:
Hi!
I implemented FilteredRawQuerySet which supports filter() and order_by() like model manager QuerySet.


Internally it contains RawQuerySet and QuerySet for the target model.

SELECT part of the query is borrowed from RawQuerySet, while WHERE and ORDER BY statements are generated via QuerySet query.

This feature is useful when the code uses programmatically generated arguments of filter() / order_by(), for example in class-based ListView with custom filtering and / or sorting.

Is such feature worth to be included to Django core?

I could just continue to use my own implementation (probably imperfect) however I am worrying about possible query generation code changes that could make my module incompatible in the future.

In case such queryset would be included to Django code, there are much more chances that the code which uses it would not break when updating to newer Django version.

Dmitriy

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.



--
Adam

Adam Johnson

unread,
Apr 21, 2017, 6:40:47 AM4/21/17
to django-d...@googlegroups.com
Joining two tables like that on one-to-one relations can be easily achieved with select_related.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/08e7fa46-8aa1-4e92-bb70-9990abfc0e4c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Adam

Dmitriy Sintsov

unread,
Apr 21, 2017, 12:16:13 PM4/21/17
to Django developers (Contributions to Django itself)
How can I chose whether select_related() will use LEFT JOIN or INNER JOIN? There are two joins, one is LEFT another is INNER one.



--
Adam

Adam Johnson

unread,
Apr 21, 2017, 12:55:31 PM4/21/17
to django-d...@googlegroups.com
It depends upon whether the foreignkey/onetoonefield is nullable.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4e96eb6a-3c22-472d-a80e-2f22ad3f49b9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Adam

Dmitriy Sintsov

unread,
Apr 21, 2017, 2:41:35 PM4/21/17
to Django developers (Contributions to Django itself)
If I understand correctly, LEFT JOIN is automatically chosen when ForeignKey(null=True) otherwise it will be INNER JOIN?

My ProfileRequisites.profile ForeignKey is not nullable yet I need to use LEFT JOIN in this query so I have profiles and ProfileRequisites in the same list. Profiles are always retrieved while ProfileRequisites only if these are available (LEFT):

class ProfileRequisites(models.Model):

    profile = models.ForeignKey(Profile, related_name='requisites')
    requisites_type = models.ForeignKey(
        ContentType, null=True, related_name='related_requisites', verbose_name='Тип реквизитов'
    )
    requisites_id = models.PositiveIntegerField(verbose_name='Ссылка на реквизиты')
    requisites_object = GenericForeignKey('requisites_type', 'requisites_id')
    is_preferred_withdrawal = models.NullBooleanField(default=None, verbose_name='Предпочтительный способ вывода')



--
Adam

Adam Johnson

unread,
Apr 21, 2017, 3:30:49 PM4/21/17
to django-d...@googlegroups.com
This isn't the place for a tutorial on how joins work in the Django ORM, there is plenty of documentation, tutorials, blog posts, and stack overflow posts for that. Check it out by internet-searching "Django select_related" or "django join". Any combination of inner and left joins is possible with the ORM.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f83009c0-047d-4114-bc12-d7260cc3e36f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Adam

Ian Foote

unread,
Apr 21, 2017, 3:44:54 PM4/21/17
to django-d...@googlegroups.com
Hi Dmitriv,

I think you're running into https://code.djangoproject.com/ticket/27332.
Unfortunately this isn't fixed yet, but there's a pull request, so it
might land in 2.0: https://github.com/django/django/pull/7560

Ian
signature.asc

Dmitriy Sintsov

unread,
Apr 21, 2017, 3:53:11 PM4/21/17
to Django developers (Contributions to Django itself)


On Friday, April 21, 2017 at 10:30:49 PM UTC+3, Adam Johnson wrote:
This isn't the place for a tutorial on how joins work in the Django ORM, there is plenty of documentation, tutorials, blog posts, and stack overflow posts for that. Check it out by internet-searching "Django select_related" or "django join". Any combination of inner and left joins is possible with the ORM.

from isp_finances.models import ProfileRequisites
ProfileRequisites.objects.select_related('profile').query.__str__()
'SELECT "isp_finances_profilerequisites"."id", "isp_finances_profilerequisites"."profile_id", "isp_finances_profilerequisites"."requisites_type_id", "isp_finances_profilerequisites"."requisites_id", "isp_finances_profilerequisites"."is_preferred_withdrawal", "isp_user_profile"."user_id", "isp_user_profile"."has_verified_email", "isp_user_profile"."is_filled", "isp_user_profile"."is_verified", "isp_user_profile"."last_name", "isp_user_profile"."first_name", "isp_user_profile"."otchestvo", "isp_user_profile"."other_names", "isp_user_profile"."birthdate", "isp_user_profile"."country_id", "isp_user_profile"."region_id", "isp_user_profile"."identity_type", "isp_user_profile"."identity_date", "isp_user_profile"."identity_numbers", "isp_user_profile"."identity_misc", "isp_user_profile"."inn", "isp_user_profile"."education", "isp_user_profile"."resume", "isp_user_profile"."rating", "isp_user_profile"."rating_reason", "isp_user_profile"."accepted_license_id" FROM "isp_finances_profilerequisites" INNER JOIN "isp_user_profile" ON ( "isp_finances_profilerequisites"."profile_id" = "isp_user_profile"."user_id" ) ORDER BY "isp_user_profile"."user_id" ASC, "isp_finances_profilerequisites"."requisites_type_id" ASC'

There is INNER JOIN generated, while I need a LEFT JOIN. I understand you want me to make ForeignKey null-able but I do not really need it.
I use select_related() (as well as prefetch_related()) in different places in my code.
 



--
Adam

Dmitriy Sintsov

unread,
Apr 21, 2017, 3:56:41 PM4/21/17
to Django developers (Contributions to Django itself)


On Friday, April 21, 2017 at 10:44:54 PM UTC+3, Ian Foote wrote:
Hi Dmitriv,

I think you're running into https://code.djangoproject.com/ticket/27332.
Unfortunately this isn't fixed yet, but there's a pull request, so it
might land in 2.0: https://github.com/django/django/pull/7560

Ian

Actually I like ORM (not only Django but any ORM) only for straightforward (not complex) queries, the more complex query become the better it looks in RAW. However improving ORM always is a good idea. If there is no need to have filtered raw QuerySet, please close my ticket.

Dmitriy Sintsov

unread,
Apr 21, 2017, 4:00:27 PM4/21/17
to Django developers (Contributions to Django itself)
There is also ListQuerySet
to be used with Prefetch() results but it's probably is too specific one.

I use it in private project.


On Friday, April 21, 2017 at 10:44:54 PM UTC+3, Ian Foote wrote:
Reply all
Reply to author
Forward
0 new messages