filtering a QuerySet after a slice

[Gary Wilson]

Is there any reason why the following is disallowed:

>>> UTPerson.objects.all()[:10].filter(name__startswith='a')
Traceback (most recent call last):
...
AssertionError: Cannot filter a query once a slice has been taken.

I would think that it should be possible as long as the QuerySet hasn't been
evaluated yet.

Gary

[Derek Anderson]

even if it was possible, i don't think it's what you would really want.
logically, this would return an indefinite number of records,
somewhere between 0 and 10, depending on what's in your table, not the
first 10 records of people who's name starts w/ an 'a'.

why are you not doing this?
UTPerson.objects.filter(name__startswith='a')[:10]

--
looking to buy or sell anything?

try: http://allurstuff.com

it's a classified ads service that
shows on a map where the seller is
(think craigslist + google maps)

plus it's 100% free :)

[Gary Wilson]

Derek Anderson wrote:
> even if it was possible, i don't think it's what you would really want.
> logically, this would return an indefinite number of records,
> somewhere between 0 and 10, depending on what's in your table, not the
> first 10 records of people who's name starts w/ an 'a'.
>
> why are you not doing this?
> UTPerson.objects.filter(name__startswith='a')[:10]

Sort of my point. Since filter() and [:] both return QuerySets why should
they be any different:

UTPerson.objects.all().filter(name__startswith='a')[:10]

UTPerson.objects.all()[:10].filter(name__startswith='a')

Also warming up for if we have a limit():

UTPerson.objects.all().filter(name__startswith='a').limit(10)
UTPerson.objects.all().limit(10).filter(name__startswith='a')

Yes, my example was a bit contrived; but think along the lines of if I had a
view that got passed the limited QuerySet as a parameter.

def view1(request):
view2(request, queryset=Model.objects.all()[:10])

def view2(request, queryset):
render_to_response('t.html', {'items': queryset.filter(...)})

Gary

[Luke Plant]

On Tuesday 04 December 2007 07:25:31 Gary Wilson wrote:

> Sort of my point. Since filter() and [:] both return QuerySets why
> should they be any different:
>
> UTPerson.objects.all().filter(name__startswith='a')[:10]

1) This, logically, should return the first ten people whose 'name'
starts with 'a'.

> UTPerson.objects.all()[:10].filter(name__startswith='a')

2) This should return the people whose 'name' starts with 'a' from the
first 10 rows in the table

This is almost certainly what most people would expect (I guess we could
argue about that...) But there is no way to specify it in SQL without
doing subselects -- you will end up doing 1) instead.

Remember that there are other methods that do not or cannot commute e.g.
trying to do order_by() after slicing -- it would change the objects
that are returned. Just because some methods do happen to commute
(filtering) doesn't mean that others should.

As an example, I would expect these 2 pieces of code to produce the same
result:

len(p for p in UTPerson.objects.all()[:10] if p.name[0] == 'a')

len(UTPerson.objects.all()[:10].filter(name__startswith='a'))

Since we can't make the 2nd match the first, we disallow it.

Luke

--
"My capacity for happiness you could fit into a matchbox without taking
out the matches first." (Marvin the paranoid android)

Luke Plant || http://lukeplant.me.uk/

[Gary Wilson]

Luke Plant wrote:
> On Tuesday 04 December 2007 07:25:31 Gary Wilson wrote:
>
>> Sort of my point. Since filter() and [:] both return QuerySets why
>> should they be any different:
>>
>> UTPerson.objects.all().filter(name__startswith='a')[:10]
>
> 1) This, logically, should return the first ten people whose 'name'
> starts with 'a'.
>
>> UTPerson.objects.all()[:10].filter(name__startswith='a')
>
> 2) This should return the people whose 'name' starts with 'a' from the
> first 10 rows in the table
>
> This is almost certainly what most people would expect (I guess we could
> argue about that...) But there is no way to specify it in SQL without
> doing subselects -- you will end up doing 1) instead.

Does that mean that most people also expect:

UTPerson.objects.filter(first_name__startswith='a').filter(last_name__startswith='b')

to return the people whose last name starts with 'b' from the people whose
first name starts with 'a'? Maybe, but filter doesn't work that way just like
slicing wouldn't work that way.

I should be able to keep piling things onto the QuerySet. You can do it with
all the other QuerySet methods that return QuerySets. Why not with the
slicing syntax (which returns a QuerySet object)?

I am just trying to spark some discussion on the matter. Personally, I
wouldn't mind if slices were changed to always return lists (like the do
currently if you use a step). And then we and in a limit() method for people
who want QuerySets returned so that they can be built up and lazily evaluated.

I see:

UTPerson.objects.all().limit(10).filter(...).limit(10)

acting like order_by() in that the second limit would override the first.

> Remember that there are other methods that do not or cannot commute e.g.
> trying to do order_by() after slicing -- it would change the objects
> that are returned. Just because some methods do happen to commute
> (filtering) doesn't mean that others should.

Well, with order_by() you can do:

UTPerson.objects.all().order_by('first_name').filter(...).order_by('last_name')

In this case, the second order_by() overrides the first.

Gary

[Rimvydas Naktinis]

I completely agree with Gary's argumentation here. And I would love to see .limit() introduced (or array slicing adapted).

I ran into a situation where I have access into intermediate queryset() construction, where I normally do add .filter() statements, but now I had to limit a number of queries returned and it caused an exception, because some other code further down the stack was modifying.

The actual query construction should be the final step, and in that case it doesn't matter when and how many times an equivalent of .limit() (or for that matter slicing) was called on the queryset. Just use the final one.

[Michael Manfre]

Array slicing doesn't always return a QuerySet and shouldn't be adapted to this proposed behavior. Directly exposing Query.set_limits() on the QuerySet would most likely have confusing behavior to at least some group of people. 

Given the current behavior of Query.set_limits(), what should be the slice start/stop for the below?

UTPerson.objects.all().set_limits(low=1, high=10).filter(...).set_limits(low=1, high=10)

If you guessed, low=1 and high=10, you'd be wrong. Subsequent calls to set_limits() "are applied relative to the existing constraints." That behavior can be changed by having the QuerySet.set_limits() call Query.clear_limits() before Query.set_limits(), but that would probably be confusing to those that are familiar with how Query.set_limits() behaves. 

If you want to be able to further filter and reapply/change the limits, you can do that already. You can't do it as chained QuerySet calls, but the only time this would make sense in code is when a QuerySet was passed with a potentially unknown can_filter state, so you'd need to reset it before using it.

if not qs.query.can_filter():

    qs.query.clear_limits()

qs.filter(...)[:10]

Regards,

Michael Manfre

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/be64f70b-4443-4c76-9ff5-db3f35fdef14%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.