Prefixing Q Objects
204 views
Skip to first unread message
Robert Schindler
Jul 10, 2019, 8:31:55 PM7/10/19
to Django developers (Contributions to Django itself)
Hi all,
I've been redirected to the mailing list from the ticked I opened [0]. Therein, I proposed to add a new method to the Q object API for prefixing Q objects with a related field's name in order to be able to create reusable Q objects.
I think the use case I pointed out might have been misunderstood. Such an API would allow writing a manager like this one, let's say for a Subscription model:
class SubscriptionManager(Manager):
def active_q(self):
return Q(cancelled=False) & Q(end_date__lt=datetime.datetime.now())
# This one is just for convenience and compatibility with the evolved practice
def active(self):
return self.filter(self.active_q())
We could then perform the following lookups:
# All active subscriptions, nothing special here
Subscription.objects.active()
# Given that a user has a ManyToManyField subscriptions, get all users with active subscriptions, without repeating any business logic
User.objects.filter(Subscription.objects.active_q().prefix("subscriptions"))
The traditional approach would be to implement methods for filtering the queryset on both the managers for Subscription and User and thus duplicating logic, or use sub-queries like so:
User.objects.filter(subscriptions__in=Subscription.objects.active())
which is clearly not wanted because of performance.
What are your opinions?
Best regards
Robert
Matthew Pava
Jul 10, 2019, 10:14:11 PM7/10/19
to django-d...@googlegroups.com
I had a use for such a feature, though I used an implementation more specific to my application.
Basically, I have a model and then I have several other models that are directly related to that model. So in the base model I would have a field to determine if the instance was marked as deleted. I don’t have this field in the sub-models, so I would pass in the related name of the base model to create a queryset of all instances of the submodels that should be marked as deleted. Actually, that’s not exactly what I was doing, but the concept is there. Saying that, I am only doing that on one field on one model in my project.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
django-develop...@googlegroups.com.
To post to this group, send email to
django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/30020900-7a4a-442e-9cce-af3d75a27c15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Robert Schindler
Jul 11, 2019, 3:54:37 AM7/11/19
to django-d...@googlegroups.com
Hi Matthew,
Am 11.07.19 um 04:13 schrieb Matthew Pava:
I'm using it for filtering a model's QuerySet for those objects a user
is authorized to see, which has to be implemented for all models in the
project so that a custom view mixin can then filter the queryset
regardless of the model. However, these queries heavily depend on
related fields.
For convenience, I created this module [0], which makes declaring the
query logic as part of a model's manager really straightforward and also
provides the well-known API for getting filtered querysets (aka
Fruit.objects.green()).
Actually, this helped me a lot declaring the needed queries DRY-free.
[0] https://github.com/efficiosoft/QProvider/blob/master/QProvider.py
Am 11.07.19 um 04:13 schrieb Matthew Pava:
> I had a use for such a feature, though I used an implementation more
> specific to my application.
>
> Basically, I have a model and then I have several other models that are
> directly related to that model. So in the base model I would have a
> field to determine if the instance was marked as deleted. I don’t have
> this field in the sub-models, so I would pass in the related name of the
> base model to create a queryset of all instances of the submodels that
> should be marked as deleted. Actually, that’s not exactly what I was
> doing, but the concept is there. Saying that, I am only doing that on
> one field on one model in my project.
Yes, that would be a use-case indeed.
> specific to my application.
>
> Basically, I have a model and then I have several other models that are
> directly related to that model. So in the base model I would have a
> field to determine if the instance was marked as deleted. I don’t have
> this field in the sub-models, so I would pass in the related name of the
> base model to create a queryset of all instances of the submodels that
> should be marked as deleted. Actually, that’s not exactly what I was
> doing, but the concept is there. Saying that, I am only doing that on
> one field on one model in my project.
I'm using it for filtering a model's QuerySet for those objects a user
is authorized to see, which has to be implemented for all models in the
project so that a custom view mixin can then filter the queryset
regardless of the model. However, these queries heavily depend on
related fields.
For convenience, I created this module [0], which makes declaring the
query logic as part of a model's manager really straightforward and also
provides the well-known API for getting filtered querysets (aka
Fruit.objects.green()).
Actually, this helped me a lot declaring the needed queries DRY-free.
[0] https://github.com/efficiosoft/QProvider/blob/master/QProvider.py
Robert Schindler
Jul 11, 2019, 12:28:59 PM7/11/19
to django-d...@googlegroups.com
Hi again,
Am 11.07.19 um 09:54 schrieb Robert Schindler:
Best regards
Robert
[0] https://github.com/efficiosoft/django-qprovider
Am 11.07.19 um 09:54 schrieb Robert Schindler:
> For convenience, I created this module [0], which makes declaring the
> query logic as part of a model's manager really straightforward and also
> provides the well-known API for getting filtered querysets (aka
> Fruit.objects.green()).
Sorry for the dead link, here [0] is the right one.
> query logic as part of a model's manager really straightforward and also
> provides the well-known API for getting filtered querysets (aka
> Fruit.objects.green()).
Best regards
Robert
[0] https://github.com/efficiosoft/django-qprovider
Shai Berger
Jul 11, 2019, 12:57:17 PM7/11/19
to 'Robert Schindler' via Django developers (Contributions to Django itself)
Hi Robert,
On Wed, 10 Jul 2019 12:13:08 -0700 (PDT)
Robert Schindler wrote:
> The traditional approach would be to implement methods for filtering
> the queryset on both the managers for Subscription and User and thus
> duplicating logic, or use sub-queries like so:
>
> User.objects.filter(subscriptions__in=Subscription.objects.active())
>
> which is clearly not wanted because of performance.
>
Not so clearly. Querysets are lazy, so the above actually only performs
one database roundtrip, and whether a join is more or less performant
than an equivalent query-with-a-sub-query can depend on many details --
mostly details of the database engine implementations. If database
engines were ideal, equivalent queries would have equivalent
performance.
More generally: The idea does seem interesting, but note that the
prefix() method doesn't need to be a member of the Q class -- it seems
to be just as easy to write an external function to do the same thing:
def add_prefix(q_obj, prefix):
"""Recursively copies the Q object, prefixing all lookup keys.
The prefix and the existing filter key are delimited by the
lookup separator __. Use this feature to delegate existing
query constraints to a related field. """
return Q(
*(
add_prefix(child, prefix)
if isinstance(child, Q)
else (prefix + LOOKUP_SEP + child[0], child[1])
for child in q_obj.children
),
_connector=q_obj.connector,
_negated=q_obj.negated,
)
So, this can be added as a utility function to your project, or even
published in a package on PyPI, without being in Django.
Given this, the question is no longer just "is this useful", but rather
"does this promote use-patterns we wish to encourage". On the ticket,
Mariusz gave an example of a possible use-case we'd actually want to
discourage. Here on the list, better use-cases were shown.
One use case you didn't bring up is using a Q object with prefix as a
"template" -- consider querying for the "first Tuesday of the month";
if there's a date field `d`, it would need to be something like
...filter(d__week_day=3, d__day__lte=7)
But with a prefix, we could say something like
def is_first_tuesday(field)
return Q(week_day=3, day__lte=7).prefix(field)
and then use it in
... filter(is_first_tuesday('d'),...)
Which, at a very first glance, looks like an excellent use-case. Except
that Django already has a way to support this, using custom lookups,
which blend better into the QuerySet API -- and so, have technical
advantages beyond the fact that they already exist and "There should be
one obvious way to do it".
So, I, at least, remain unconvinced that overall, this is an
improvement to Django's API.
Shai
On Wed, 10 Jul 2019 12:13:08 -0700 (PDT)
Robert Schindler wrote:
> The traditional approach would be to implement methods for filtering
> the queryset on both the managers for Subscription and User and thus
> duplicating logic, or use sub-queries like so:
>
> User.objects.filter(subscriptions__in=Subscription.objects.active())
>
> which is clearly not wanted because of performance.
>
one database roundtrip, and whether a join is more or less performant
than an equivalent query-with-a-sub-query can depend on many details --
mostly details of the database engine implementations. If database
engines were ideal, equivalent queries would have equivalent
performance.
More generally: The idea does seem interesting, but note that the
prefix() method doesn't need to be a member of the Q class -- it seems
to be just as easy to write an external function to do the same thing:
def add_prefix(q_obj, prefix):
"""Recursively copies the Q object, prefixing all lookup keys.
The prefix and the existing filter key are delimited by the
lookup separator __. Use this feature to delegate existing
query constraints to a related field. """
return Q(
*(
add_prefix(child, prefix)
if isinstance(child, Q)
else (prefix + LOOKUP_SEP + child[0], child[1])
for child in q_obj.children
),
_connector=q_obj.connector,
_negated=q_obj.negated,
)
So, this can be added as a utility function to your project, or even
published in a package on PyPI, without being in Django.
Given this, the question is no longer just "is this useful", but rather
"does this promote use-patterns we wish to encourage". On the ticket,
Mariusz gave an example of a possible use-case we'd actually want to
discourage. Here on the list, better use-cases were shown.
One use case you didn't bring up is using a Q object with prefix as a
"template" -- consider querying for the "first Tuesday of the month";
if there's a date field `d`, it would need to be something like
...filter(d__week_day=3, d__day__lte=7)
But with a prefix, we could say something like
def is_first_tuesday(field)
return Q(week_day=3, day__lte=7).prefix(field)
and then use it in
... filter(is_first_tuesday('d'),...)
Which, at a very first glance, looks like an excellent use-case. Except
that Django already has a way to support this, using custom lookups,
which blend better into the QuerySet API -- and so, have technical
advantages beyond the fact that they already exist and "There should be
one obvious way to do it".
So, I, at least, remain unconvinced that overall, this is an
improvement to Django's API.
Shai
Robert Schindler
Jul 11, 2019, 1:20:47 PM7/11/19
to django-d...@googlegroups.com
Hi Shai,
Am 11.07.19 um 18:57 schrieb Shai Berger:
what I often heard, but I didn't take any benchmarks myself.
> More generally: The idea does seem interesting, but note that the
> prefix() method doesn't need to be a member of the Q class -- it seems
> to be just as easy to write an external function to do the same thing:
Sure it is, but I, personally, find it more appealing to have the
functionality as part of the Q object itself, because it operates on a
single Q object as input and returns another one, which makes this an
ideal candidate for an object member, imho. Not to say that it must be
in Django directly.
unittests and publish the package, the structure is already there in the
repo.
> So, I, at least, remain unconvinced that overall, this is an
> improvement to Django's API.
I agree that this makes most sense in conjunction with the additional
QProvider class for defining hybrid Q and QuerySet generators on
managers, which, even though very short and simple, is another addition
to be considered in this regard.
Thanks for taking the time.
Best regards
Robert
Am 11.07.19 um 18:57 schrieb Shai Berger:
> Not so clearly. Querysets are lazy, so the above actually only performs
> one database roundtrip, and whether a join is more or less performant
> than an equivalent query-with-a-sub-query can depend on many details --
> mostly details of the database engine implementations. If database
> engines were ideal, equivalent queries would have equivalent
> performance.
I'm no database expert and you are probably right. At least that was
> one database roundtrip, and whether a join is more or less performant
> than an equivalent query-with-a-sub-query can depend on many details --
> mostly details of the database engine implementations. If database
> engines were ideal, equivalent queries would have equivalent
> performance.
what I often heard, but I didn't take any benchmarks myself.
> More generally: The idea does seem interesting, but note that the
> prefix() method doesn't need to be a member of the Q class -- it seems
> to be just as easy to write an external function to do the same thing:
functionality as part of the Q object itself, because it operates on a
single Q object as input and returns another one, which makes this an
ideal candidate for an object member, imho. Not to say that it must be
in Django directly.
>
> So, this can be added as a utility function to your project, or even
> published in a package on PyPI, without being in Django.
That's the way I planned to do it. When I like the API, I'm going to add
> So, this can be added as a utility function to your project, or even
> published in a package on PyPI, without being in Django.
unittests and publish the package, the structure is already there in the
repo.
>
> Given this, the question is no longer just "is this useful", but rather
> "does this promote use-patterns we wish to encourage". On the ticket,
> Mariusz gave an example of a possible use-case we'd actually want to
> discourage. Here on the list, better use-cases were shown.
>
> One use case you didn't bring up is using a Q object with prefix as a
> "template" -- consider querying for the "first Tuesday of the month";
> if there's a date field `d`, it would need to be something like
>
> ...filter(d__week_day=3, d__day__lte=7)
>
> But with a prefix, we could say something like
>
> def is_first_tuesday(field)
> return Q(week_day=3, day__lte=7).prefix(field)
>
> and then use it in
>
> ... filter(is_first_tuesday('d'),...)
>
> Which, at a very first glance, looks like an excellent use-case. Except
> that Django already has a way to support this, using custom lookups,
> which blend better into the QuerySet API -- and so, have technical
> advantages beyond the fact that they already exist and "There should be
> one obvious way to do it".
Looks interesting, haven't thought of such a use case myself yet.
> Given this, the question is no longer just "is this useful", but rather
> "does this promote use-patterns we wish to encourage". On the ticket,
> Mariusz gave an example of a possible use-case we'd actually want to
> discourage. Here on the list, better use-cases were shown.
>
> One use case you didn't bring up is using a Q object with prefix as a
> "template" -- consider querying for the "first Tuesday of the month";
> if there's a date field `d`, it would need to be something like
>
> ...filter(d__week_day=3, d__day__lte=7)
>
> But with a prefix, we could say something like
>
> def is_first_tuesday(field)
> return Q(week_day=3, day__lte=7).prefix(field)
>
> and then use it in
>
> ... filter(is_first_tuesday('d'),...)
>
> Which, at a very first glance, looks like an excellent use-case. Except
> that Django already has a way to support this, using custom lookups,
> which blend better into the QuerySet API -- and so, have technical
> advantages beyond the fact that they already exist and "There should be
> one obvious way to do it".
> So, I, at least, remain unconvinced that overall, this is an
> improvement to Django's API.
QProvider class for defining hybrid Q and QuerySet generators on
managers, which, even though very short and simple, is another addition
to be considered in this regard.
Thanks for taking the time.
Best regards
Robert
Robert Schindler
Jul 11, 2019, 1:24:15 PM7/11/19
to django-d...@googlegroups.com
Am 11.07.19 um 19:20 schrieb Robert Schindler:
>>
Best regards
Robert
> Sure it is, but I, personally, find it more appealing to have the
> functionality as part of the Q object itself, because it operates on a
> single Q object as input and returns another one, which makes this an
> ideal candidate for an object member, imho. Not to say that it must be
> in Django directly.
Oops, should have been: "I don't want to say it must be in Django directly."
> functionality as part of the Q object itself, because it operates on a
> single Q object as input and returns another one, which makes this an
> ideal candidate for an object member, imho. Not to say that it must be
> in Django directly.
>>
Best regards
Robert
Reply all
Reply to author
Forward
0 new messages