DB API - the limiting syntax, is it magic?

5 views
Skip to first unread message

Mike Scott

unread,
Nov 30, 2007, 10:41:06 PM11/30/07
to django-d...@googlegroups.com
Hi all,

I was in two minds about writing this as a ticket, mainly because I understand that it is something that is a design decision.

Okay so the issue I came across today was that one of my python developers changed some syntax on one of my views with loads some data from the database, this particular model was being filtered, and limited. However due to the fact that the limit syntax was using pythons slicing syntax, and not a limit method, he thought that it wouldn't affect db (I have since showed him that the table in question has 3 million+ rows, and that grows by a few thousand every night). However it does raise and interesting point - should the limit syntax be apart of a method, rather than hijack the python syntax, is it too magic?

I'm very open to opinions here, as it is both clearly labeled in the Documentation and has stuck so long, but it was an interesting development for a python developer to come into Django for the first time and discover this.

Regards,


Mike Scott

fauxm...@gmail.com

unread,
Nov 30, 2007, 11:00:20 PM11/30/07
to Django developers
To clarify, code such as the following can be a little confusing:

p = MyModel.objects.filter(foo__id = self.id).order_by('-timestamp')[:
1]
if p:
return p[0]

To a python developer who is unfamiliar with django's magic limiting
syntax, the slice there looks unnecessary.

Writing objects.filter(...).order_by(...).limit(1) would be clearer
and significantly less error prone.

SmileyChris

unread,
Dec 1, 2007, 5:04:09 AM12/1/07
to Django developers
On Dec 1, 5:00 pm, "sam.bi...@aquis.co.nz" <fauxmon...@gmail.com>
wrote:
> To a python developer who is unfamiliar with django's magic limiting
> syntax, the slice there looks unnecessary.

If/when we get a __nonzero__ method, it will be unnecessary.

Malcolm Tredinnick

unread,
Dec 2, 2007, 9:31:12 AM12/2/07
to django-d...@googlegroups.com

No it wouldn't. It would be differently error-prone.

Using Pythonic syntax is nice and it's pretty rare that adding the slice
is really going to affect performance, since it only queries the
database if you haven't *already* run the database query (if you slice a
queryset that is already cached, it pulls the results out of the cache).

I'd be -1 on changing this.

Malcolm


Adrian Holovaty

unread,
Dec 3, 2007, 12:28:29 AM12/3/07
to django-d...@googlegroups.com
On Dec 2, 2007 8:31 AM, Malcolm Tredinnick <mal...@pointy-stick.com> wrote:
> On Fri, 2007-11-30 at 20:00 -0800, sam....@aquis.co.nz wrote:
> > To clarify, code such as the following can be a little confusing:
> >
> > p = MyModel.objects.filter(foo__id = self.id).order_by('-timestamp')[:
> > 1]
> > if p:
> > return p[0]
> >
> > To a python developer who is unfamiliar with django's magic limiting
> > syntax, the slice there looks unnecessary.
> >
> > Writing objects.filter(...).order_by(...).limit(1) would be clearer
> > and significantly less error prone.
>
> No it wouldn't. It would be differently error-prone.

Frankly, I have found the slicing syntax hard to understand and
error-prone myself. I think one of the reasons it seems magic is that
it's one of the *rare* cases in which we use a Python
"magic-syntax-ism" (like operator overloading, for instance) in the
framework, plus the fact that, what, *every* other piece of QuerySet
functionality uses a method. It seems bolted on.

I'd be +1 on adding a distinct limit() method, and keeping the legacy
list method for masochists who actually enjoy coding in that way.

Adrian

--
Adrian Holovaty
holovaty.com | djangoproject.com

James Bennett

unread,
Dec 3, 2007, 12:50:21 AM12/3/07
to django-d...@googlegroups.com
On 12/2/07, Adrian Holovaty <holo...@gmail.com> wrote:
> Frankly, I have found the slicing syntax hard to understand and
> error-prone myself.

I guess I'm curious as to what's difficult or error-prone about it;
I've never run into a problem where slicing was the cuplrit, and it
feels like a more intuitive and Pythonic syntax than the "limit" and
"offset" arguments we used to have in the pre-magic-removal days.

> I think one of the reasons it seems magic is that
> it's one of the *rare* cases in which we use a Python
> "magic-syntax-ism" (like operator overloading, for instance) in the
> framework, plus the fact that, what, *every* other piece of QuerySet
> functionality uses a method. It seems bolted on.

In a way this feels like the same sort of debate as the one over
whether to implement __nonzero__() as a way of checking for an empty
QuerySet, or to use a dedicated exists() method; going one way is more
"Pythonic", going the other way is more "SQL-ish". I'd prefer to stick
to one way to do things, and probably it'd be a good idea to have a
design discussion about which way Django will lean on these issues.


--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."

James Bennett

unread,
Dec 3, 2007, 1:13:49 AM12/3/07
to django-d...@googlegroups.com
On 12/2/07, Adrian Holovaty <holo...@gmail.com> wrote:
> Frankly, I have found the slicing syntax hard to understand and
> error-prone myself. I think one of the reasons it seems magic is that
> it's one of the *rare* cases in which we use a Python
> "magic-syntax-ism" (like operator overloading, for instance) in the
> framework, plus the fact that, what, *every* other piece of QuerySet
> functionality uses a method. It seems bolted on.
>
> I'd be +1 on adding a distinct limit() method, and keeping the legacy
> list method for masochists who actually enjoy coding in that way.

And.. it's late and I'm possibly not thinking clearly, but I feel like
fleshing out what I've already said, because this is something I have
a strong opinion on.

Let's step back from the case at hand and consider something else so
it'll be a bit more objective to look at. That means another "magic"
Python method, so I'll pick the __str__()/__unicode__() combo we do on
model classes.

To someone who's not familiar with Python, the fact that defining
these methods will cause a complex object to -- in certain
circumstances -- "magically" become a string is undoubtedly confusing.
But the solution to that isn't to stop using them, it's to help people
become familiar with the conventions of Python programming; if we
switched to, say, defining 'toUnicodeString()' and 'toByteString()' on
models, a lot more people would be able to immediately pick up Django
and grok what's going on, but we'd be going against the grain of
Python, which provides __unicode__() and __str__() for these purposes
already, and we'd be confusing people who are familiar with Python and
who expect that "convert Object X to a string" is going to call
__unicode__() or __str__().

I don't think there's anybody here who'd advocate adding explicit
string-generating methods on models just for the sake of letting
people avoid the need to understand how Python works; we'd be working
against the language in a painful way, and in the long run it'd hurt a
lot more than it helped.

Getting back to the issue at hand, slicing a QuerySet is the same
situation: a lot of people would undoubtedly grok things more quickly
if there were explicit "limit()", "offset()", etc. methods on
QuerySet, but Python already provides a standard language feature for
objects which want to emulate a sequence and allow users to retrieve
only a portion of the sequence, and defining our own separate set of
methods for this would be working against the language in a big way,
and we'd be doing it, essentially, so that people don't have to learn
how Python works. Which is a disservice to everyone who uses Django:

1. Experienced Python programmers would wonder why the hell we're not
just using the features Python provides for this use case.
2. Inexperienced Python programmers would just get bitten the first
time they tried to slice some other sequence type and be right back in
the same boat they were in originally.

And, ultimately, referring to a standard language feature as "magic"
does a large disservice both to Python and to people who are trying to
learn Django and Python: the slicing syntax, and the way it's
implemented, is no more or less "magical" than, say, a Java class
which implements standard interfaces out of java.util or java.lang.

So if it's possible to have a negative vote which involves aleph
numbers or other such suitably infinite quantities, that's how I'd
vote on this change.

Mike Scott

unread,
Dec 3, 2007, 2:21:43 AM12/3/07
to django-d...@googlegroups.com
James, I'm think what I'm getting at more is not the fact that its "magical", maybe that is the wrong choice of word. But my opinion is more of the fact that it doesn't conform to the rest of the django database commands.

I do think you put it aptly in asking do we want to be more SQL-style or  Python-style. Ultimately it boils down to people understanding the framework better, and keeping it as is may infact do that. I think it is something that needs to be considered quite carefully. Promoting slicing as the recommended way, but still allowing a limit() function in there to conform with the rest of the db api. Conformity, to a set standard (which in this case doesn't seem to exist - which is another reason I am +1 on the DEP idea), is better than doing it just cause you can.

What the issue was (and the reason I bought this up) was that it was understood that the python functionality was executed after the method was, and therefor there was an understanding that it wouldn't make any difference:

Models.objects.filter(date__gt=date_start)[:1]

However, it did and it would have been clearer as:

Model.objects.filter(date__gt=date_start).limit(1)

Its something that has been solved by marking with a comment, however I'm firmly a fence sitter at this point, so I'm ready to be swung either way. But I don't think your comments so far have done so.


Mike.

James Bennett

unread,
Dec 3, 2007, 2:53:14 AM12/3/07
to django-d...@googlegroups.com
On 12/3/07, Mike Scott <mic...@gmail.com> wrote:
> James, I'm think what I'm getting at more is not the fact that its
> "magical", maybe that is the wrong choice of word. But my opinion is more of
> the fact that it doesn't conform to the rest of the django database
> commands.

I suppose it depends on what you mean by "conforms"; if we wanted to
run with that, I'd ask why you can iterate directly over a QuerySet in
a for loop instead of doing something like, say

queryset = MyModel.objects.all()
for obj in queryset.iterate():
print obj


After all, iteration is implemented using a special Python feature
just the same as slicing, and it's not necessarily obvious that a
QuerySet is something you ought to be able to iterate over without
calling a special method.

> I do think you put it aptly in asking do we want to be more SQL-style or
> Python-style. Ultimately it boils down to people understanding the framework
> better, and keeping it as is may infact do that. I think it is something
> that needs to be considered quite carefully. Promoting slicing as the
> recommended way, but still allowing a limit() function in there to conform
> with the rest of the db api. Conformity, to a set standard (which in this
> case doesn't seem to exist - which is another reason I am +1 on the DEP
> idea), is better than doing it just cause you can.

I'm very much against maintaining two separate ways to do the same
thing. And I'm very strongly in favor of using the "Pythonic" solution
as opposed to the "SQL" solution.

> What the issue was (and the reason I bought this up) was that it was
> understood that the python functionality was executed after the method was,
> and therefor there was an understanding that it wouldn't make any
> difference:

That's the point, though, where it's important to know how Python's
slicing syntax is applied to custom classes, and to go look to see how
the slicing is implemented. We could do a better job of documenting
exactly what slicing does (since it translates into LIMIT/OFFSET in
the DB query), but that still doesn't strike me as an argument for
killing off what is, essentially, the standard Python syntax for this
sort of operation.

> Its something that has been solved by marking with a comment, however I'm
> firmly a fence sitter at this point, so I'm ready to be swung either way.
> But I don't think your comments so far have done so.

After I wrote my second reply to Adrian, I decided to really go
all-out and just write a blog entry about this; my main beef is with
calling things "magic" when really they're just standard Python
features, but maybe it'll help shed some light on the larger
discussion:

http://www.b-list.org/weblog/2007/dec/03/magic/

Malcolm Tredinnick

unread,
Dec 3, 2007, 10:35:27 AM12/3/07
to django-d...@googlegroups.com

Adding limit() -- which will also require an offset() method -- is just
as error prone and is less intuitive than the current system (which
conforms to normal Python behaviour). All over the world conversations
such as the following will occur:

Jim:
I want to select the third, fourth and fifth items from
this queryset. What's the best way to do that? I always
forget the syntax.

Jane:
I think you limit() your queryset to five items and then
offset() to the third and it will work. You could also
offset() first to item three and then limit() to three.

Jim:
Really, don't limit() and offset() commute?

Jane:
Well, all the other queryset methods apply
incrementally, so these two should as well, shouldn't
they? In the latter case you first apply the offset,
which restricts the result set and *then* limit it
further. offset(3).limit(5) would give you items three
through seven, wouldn't it?

Jim:
But it's deliberately written as a leaky abstraction of
the SQL extensions called limit and offset. So why
doesn't it behave the same as they do? Maybe that's the
way it works. We're going to have to look it up
again. :-(

I guess it's not possible to have both, so the Django
developers just chose arbitrarily. That sucks... it's
impossible to use logic to understand this. They should
be called chop_first() and interval() then.

Jane:
Yeah, what we really need is a way to specify both at
once. A method that takes the lower and upper bound.
That would remove the ambiguity. Maybe you could leave
out one or the other. We could call it ... *thinks for a
bit* ... __getitem__()!

Jim:
Agreed. We're really taking a subsection of a sequence
here, right? Leaky abstractions aside, Django is
fundamentally presenting a Python view of the data and
only using the database as a persistence mechanism.
Slicing is the normal way to extract sections from
sequences in Python. Wonder why they didn't do that?

Ancient John (older programmer, who's been around the company
for years):
They used to do that. Then Google cut off the oxygen to
the django-developers group and people stopped thinking
clearly. Confusion spread throughout the land.
Apparently the current method is deemed superior because
it keeps people talking to each other and doesn't let
familiarity and standard patterns settle in. It's called
social engineering: you have to be more social to get
stuff done.

This is all very disappointing.

Malcolm

Luke Plant

unread,
Dec 3, 2007, 11:16:54 AM12/3/07
to django-d...@googlegroups.com
On Monday 03 December 2007 15:35:27 Malcolm Tredinnick wrote:

> Ancient John (older programmer, who's been around the company
> for years):
> They used to do that. Then Google cut off the oxygen
> to the django-developers group and people stopped thinking clearly.
> Confusion spread throughout the land. Apparently the current method
> is deemed superior because it keeps people talking to each other and
> doesn't let familiarity and standard patterns settle in. It's called
> social engineering: you have to be more social to get stuff done.

Lol, very good!

Luke

--
"Mistakes: It could be that the purpose of your life is only to serve
as a warning to others." (despair.com)

Luke Plant || http://lukeplant.me.uk/

Adrian Holovaty

unread,
Dec 3, 2007, 8:34:14 PM12/3/07
to django-d...@googlegroups.com
On Dec 2, 2007 11:50 PM, James Bennett <ubern...@gmail.com> wrote:
> I guess I'm curious as to what's difficult or error-prone about it;
> I've never run into a problem where slicing was the cuplrit, and it
> feels like a more intuitive and Pythonic syntax than the "limit" and
> "offset" arguments we used to have in the pre-magic-removal days.

Here's a specific problem that's bitten me a number of times. I want
to get a list of model objects and do something special with the first
one. I want to pass both the full list and the first one to the
template.

objects = MyModel.objects.filter(site=1)
first_one = objects[0]
do_something_special(first_one)
return render_to_response('t.html', {'objects': objects, 'first':
first_one})

Although this works, it performs two database queries -- one to
retrieve all of the objects, and one to retrieve the first one (with a
"LIMIT" clause). That's unacceptable.

The (unintuitive) way to fix the problem is to wrap the first line in list() --

objects = list(MyModel.objects.filter(site=1))

The counter argument here is, "Well, that's exactly what you have to
do with any iteratable." And my counter argument to that is, "Yeah,
but normal iterables raise TypeError when you try to slice them."

The confusion here stems from the fact that a QuerySet *sometimes*
behaves like a list, but not always. After the QuerySet has been
evaluated, then the slicing syntax simply selects items from the list,
but before the QuerySet has been evaluated, the slicing syntax acts as
a LIMIT/OFFSET. Different things happen depending on the context --
and that's what I find confusing.

> Let's step back from the case at hand and consider something else so
> it'll be a bit more objective to look at. That means another "magic"
> Python method, so I'll pick the __str__()/__unicode__() combo we do on
> model classes.

Sorry for leading you astray and apparently causing you to spend a
long time writing a blog entry about what is and isn't magic! I
shouldn't have used the word "magic," as it's a loaded term and has
different meanings in different contexts (much like QuerySet!). I'll
restate my philosophical problem with this API without using that
word. :-)

My problem with the slicing syntax is that it's the *only* part of the
QuerySet API that uses Python syntax hacking -- as in, "the plus sign
now does something different," or "the slice operator now does
something different." Everything else in the API uses normal Python
methods. It's this inconsistency that bothers me.

I'm still +1 on fixing this wart, but it won't be the end of the world
if nobody agrees with me. I just want folks to understand my
reasoning.

Ian Kelly

unread,
Dec 3, 2007, 9:20:56 PM12/3/07
to django-d...@googlegroups.com

I may be alone in this, but the slicing syntax actually seems a bit
non-Pythonic to me. Query sets are conceptually similar in many ways
to iterators, yet iterators (at least the ones in the standard
library) are not slicable. In order to do that, you have to use
itertools.islice, a function call. Similarly, to reverse an iterator,
you don't slice it with [::-1]; you call reversed, another function.

It seems to me the difference is that slicing, like attribute lookup,
is intuitively a simple operation that works on data already in
memory. If the operation is going to do something substantially more
complex than that, as QuerySet does, then implementing it using
__getitem__ or a property is just misleading; it's actually a method,
and it ought to be denoted as such.

In short, the fact that it uses a feature of Python doesn't
necessarily make it Pythonic in my view. Count me as +0 on the
change.

-root

Gary Wilson

unread,
Dec 4, 2007, 1:57:01 AM12/4/07
to django-d...@googlegroups.com
Adrian Holovaty wrote:
> On Dec 2, 2007 11:50 PM, James Bennett <ubern...@gmail.com> wrote:
>> I guess I'm curious as to what's difficult or error-prone about it;
>> I've never run into a problem where slicing was the cuplrit, and it
>> feels like a more intuitive and Pythonic syntax than the "limit" and
>> "offset" arguments we used to have in the pre-magic-removal days.
>
> Here's a specific problem that's bitten me a number of times. I want
> to get a list of model objects and do something special with the first
> one. I want to pass both the full list and the first one to the
> template.
>
> objects = MyModel.objects.filter(site=1)
> first_one = objects[0]
> do_something_special(first_one)
> return render_to_response('t.html', {'objects': objects, 'first':
> first_one})
>
> Although this works, it performs two database queries -- one to
> retrieve all of the objects, and one to retrieve the first one (with a
> "LIMIT" clause). That's unacceptable.

But what would the situation be with a new limit() method...

objects = MyModel.objects.filter(site=1)
first_one = objects.limit(1)


do_something_special(first_one)
return render_to_response('t.html', {'objects': objects, 'first': first_one})

If limit() returns a new QuerySet, you're still going to have two queries.

Now, maybe we could introduce some mechanics into QuerySet so that your new
QuerySet returned by limit() would keep a reference to the original QuerySet
to use for slicing it's cached results (evaluating it if it hasn't already
been evaluated). Of course, something like this would also fix the two
queries with slicing :)


On a similar note, we have two-query situations happening with count() and
values() too...

objects = MyModel.objects.filter(site=1)
count = objects.count()

objects = MyModel.objects.filter(site=1)
skinny_objects = objects.values('field1')

both of which could use an optimization if their original QuerySet has already
been evaluated.

> The (unintuitive) way to fix the problem is to wrap the first line in list() --
>
> objects = list(MyModel.objects.filter(site=1))

Another option might be to make slicing return a list and not a QuerySet,
which is already happening if you use a step with your slice.

> The counter argument here is, "Well, that's exactly what you have to
> do with any iteratable." And my counter argument to that is, "Yeah,
> but normal iterables raise TypeError when you try to slice them."

And I say QuerySets are more like lazy-evaluated lists. lists are iterable
and can be sliced.

> The confusion here stems from the fact that a QuerySet *sometimes*
> behaves like a list, but not always. After the QuerySet has been
> evaluated, then the slicing syntax simply selects items from the list,
> but before the QuerySet has been evaluated, the slicing syntax acts as
> a LIMIT/OFFSET. Different things happen depending on the context --
> and that's what I find confusing.

I don't see how this fact would lead to confusion as this is only how things
work behind the scenes. To the developer using a QuerySet, it behaves like a
list no matter if the QuerySet has been evaluated or not. I do agree however,
that we be efficient as possible on the backend.

> My problem with the slicing syntax is that it's the *only* part of the
> QuerySet API that uses Python syntax hacking -- as in, "the plus sign
> now does something different," or "the slice operator now does
> something different." Everything else in the API uses normal Python
> methods. It's this inconsistency that bothers me.
>
> I'm still +1 on fixing this wart, but it won't be the end of the world
> if nobody agrees with me. I just want folks to understand my
> reasoning.

I wouldn't be against adding limit() and offset() to _complement_ the existing
slicing capability. So call me +0 for adding limit() and offset() and -1 for
removing slicing support.

Gary

Wolfram Kriesing

unread,
Dec 4, 2007, 3:53:15 AM12/4/07
to django-d...@googlegroups.com
I remember my beginner times with django, i was searching for limit()
until I had to remind myself multiple times that you have to use [x:y] for it.
limit() seemed the reasonable straightforward thing ... and thats what
i am used to from python
but this time it was different

Providing limit() would be consistent to the rest of the querying API,
since all
query building is done using function calls!

my 2 cents

wolfram

--
cu

Wolfram

Michael Radziej

unread,
Dec 4, 2007, 4:00:18 AM12/4/07
to django-d...@googlegroups.com
On Tue, Dec 04, Gary Wilson wrote:

> And I say QuerySets are more like lazy-evaluated lists. lists are iterable
> and can be sliced.

I think this hits the point.

What do you think about this: a QuerySet fetches a chunk of elements, say
100, as soon as you request at least one? So aQuerySet[0] could fetch the
first 100 or so elements behind the scene and return the first one, and the
next access would not have to go to the database.

Michael

--
noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg -
Tel +49-911-9352-0 - Fax +49-911-9352-100
http://www.noris.de - The IT-Outsourcing Company

Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk -
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689

Wolfram Kriesing

unread,
Dec 4, 2007, 4:08:09 AM12/4/07
to django-d...@googlegroups.com
On Dec 4, 2007 10:00 AM, Michael Radziej <m...@noris.de> wrote:
>
> On Tue, Dec 04, Gary Wilson wrote:
>
> > And I say QuerySets are more like lazy-evaluated lists. lists are iterable
> > and can be sliced.
>
> I think this hits the point.
>
> What do you think about this: a QuerySet fetches a chunk of elements, say
> 100, as soon as you request at least one? So aQuerySet[0] could fetch the
> first 100 or so elements behind the scene and return the first one, and the
> next access would not have to go to the database.

-1, if the query is a more complex and you even hit an unoptimized query
this will have an effect on the overall performance, since every request
does this!

wolfram

> Michael
>
> --
> noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg -
> Tel +49-911-9352-0 - Fax +49-911-9352-100
> http://www.noris.de - The IT-Outsourcing Company
>
> Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk -
> Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689
>
>
> >
>

--
cu

Wolfram

Wolfram Kriesing

unread,
Dec 4, 2007, 9:41:07 AM12/4/07
to django-d...@googlegroups.com
another thing i jsut ran into:

this causes an IndexError
Model.objects.filter(something)[0]
you have to add the colon before the 0
Model.objects.filter(something)[:0]
the limit(0) would not bring this error, it would not let
first-time-users run into this problem

another 2 cents

wolfram

--
cu

Wolfram

Luke Plant

unread,
Dec 4, 2007, 9:57:00 AM12/4/07
to django-d...@googlegroups.com
On Tuesday 04 December 2007 14:41:07 Wolfram Kriesing wrote:
> another thing i jsut ran into:
>
> this causes an IndexError
> Model.objects.filter(something)[0]
> you have to add the colon before the 0
> Model.objects.filter(something)[:0]
> the limit(0) would not bring this error, it would not let
> first-time-users run into this problem

I'm not sure why [:0], would ever be useful -- it always returns an
empty list, right? Did you mean [:1] ? But anyway, the behaviour here
is exactly consistent with python lists:

>>> [][0]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list index out of range
>>> [][:0]
[]
>>> [][:1]
[]


We are not helping newbies by making slicing for QuerySets work
differently -- in fact we've gone to some length to make sure it
behaves as analogously as possible -- it throws the same exceptions
etc.

Luke

--
"My capacity for happiness you could fit into a matchbox without taking
out the matches first." (Marvin the paranoid android)

Luke Plant || http://lukeplant.me.uk/

Wolfram Kriesing

unread,
Dec 4, 2007, 10:06:28 AM12/4/07
to django-d...@googlegroups.com
you are right!
lol, trapped myself nicely ... happens when you dont think :-)

thx

wolfram

--
cu

Wolfram

Adrian Holovaty

unread,
Dec 4, 2007, 11:18:11 AM12/4/07
to django-d...@googlegroups.com
On Dec 4, 2007 12:57 AM, Gary Wilson <gary....@gmail.com> wrote:
> But what would the situation be with a new limit() method...
>
> objects = MyModel.objects.filter(site=1)
> first_one = objects.limit(1)
> do_something_special(first_one)
> return render_to_response('t.html', {'objects': objects, 'first': first_one})
>
> If limit() returns a new QuerySet, you're still going to have two queries.

No -- I guess I didn't explain myself well enough. In this case, I
wouldn't use limit(). I have two goals:

* Retrieve all objects in the table.
* Do something special with the first one (once the whole list has
been retrieved).

The ideal API would look like this, and it would only run a single query:

objects = MyModel.objects.filter(site=1)
first_one = objects[0]

Adrian

Patryk Zawadzki

unread,
Dec 4, 2007, 11:21:28 AM12/4/07
to django-d...@googlegroups.com
2007/12/4, Adrian Holovaty <holo...@gmail.com>:

> No -- I guess I didn't explain myself well enough. In this case, I
> wouldn't use limit(). I have two goals:
>
> * Retrieve all objects in the table.
> * Do something special with the first one (once the whole list has
> been retrieved).
>
> The ideal API would look like this, and it would only run a single query:
>
> objects = MyModel.objects.filter(site=1)
> first_one = objects[0]

Maybe introduce a resolve() or results() call that just returns list(self)?

--
Patryk Zawadzki
PLD Linux Distribution

Forest Bond

unread,
Dec 4, 2007, 11:25:42 AM12/4/07
to django-d...@googlegroups.com
Hi,

Isn't that sort of wanting it both ways -- wanting querysets to be both lazy and
not? Surely if the caller wants the objects fetched *now*, he should have to
indicate that, right? Is this not a reasonable way to do that?

objects = list(MyModel.objects.filter(site=1))
first_one = objects[0]

Maybe another method would make that read nicer:

objects = MyModel.objects.filter(site=1).fetch()
first_one = objects[0]

-Forest
--
Forest Bond
http://www.alittletooquiet.net

signature.asc

Forest Bond

unread,
Dec 4, 2007, 11:28:40 AM12/4/07
to django-d...@googlegroups.com
Hi,

Replying to self.

On Tue, Dec 04, 2007 at 11:25:42AM -0500, Forest Bond wrote:
> objects = MyModel.objects.filter(site=1).fetch()
> first_one = objects[0]

Incidentally, qs.fetch() need not be equivalent to list(qs). It could just
trigger qs to cache the results of the query and return qs itself. That would
ensure that the query is only executed once, unless further manipulations are
attempted.

Not that that is necessarily the right thing to do.

signature.asc

David Cramer

unread,
Dec 4, 2007, 9:00:31 PM12/4/07
to Django developers
Slicing is cool, but I'm +1 for deprecating it and using .limit().
Slicing in my mind should return an iterable and you shouldn't be
messing w/ properties on that iterable, even though somethings you
might want to.

For example:

We extend the queryset model for our fulltext search -- which uses
slicing to do pages. But there's more information we could add, such
as total results, max results, etc, to the queryset result model. You
could do blah = mysearchquery[0:10], and then blah.someattr, but it
just doesn't feel right. Whereas doing blah = mysearchquery.limit(0,
10), and blah.attr actually makes sense.
> signature.asc
> 1KDownload

Gary Wilson

unread,
Dec 5, 2007, 12:31:39 AM12/5/07
to django-d...@googlegroups.com
Adrian Holovaty wrote:
> On Dec 4, 2007 12:57 AM, Gary Wilson <gary....@gmail.com> wrote:
>> But what would the situation be with a new limit() method...
>>
>> objects = MyModel.objects.filter(site=1)
>> first_one = objects.limit(1)
>> do_something_special(first_one)
>> return render_to_response('t.html', {'objects': objects, 'first': first_one})
>>
>> If limit() returns a new QuerySet, you're still going to have two queries.
>
> No -- I guess I didn't explain myself well enough. In this case, I
> wouldn't use limit(). I have two goals:

Ah, my fault. You didn't mention limit() in your post, I was just reading
through from your post a couple back in the thread.

> * Retrieve all objects in the table.
> * Do something special with the first one (once the whole list has
> been retrieved).
>
> The ideal API would look like this, and it would only run a single query:
>
> objects = MyModel.objects.filter(site=1)
> first_one = objects[0]

Any particular ideas about the implementation?

1) Change slices to always return lists and introduce limit() and offset()
methods that return QuerySets.

2) objects[0] evaluates MyModel.objects.filter(site=1) and then returns the
first item.

3) Try to be smart and do a SELECT * LIMIT 1 if first_one is used first, then
run SELECT * when objects is used. Or, if objects is used first run SELECT *,
then when first_one is used it just take the first item of the result cache of
the QuerySet it cloned.

others ideas?

Gary

Reply all
Reply to author
Forward
0 new messages