No it wouldn't. It would be differently error-prone.
Using Pythonic syntax is nice and it's pretty rare that adding the slice
is really going to affect performance, since it only queries the
database if you haven't *already* run the database query (if you slice a
queryset that is already cached, it pulls the results out of the cache).
I'd be -1 on changing this.
Malcolm
Frankly, I have found the slicing syntax hard to understand and
error-prone myself. I think one of the reasons it seems magic is that
it's one of the *rare* cases in which we use a Python
"magic-syntax-ism" (like operator overloading, for instance) in the
framework, plus the fact that, what, *every* other piece of QuerySet
functionality uses a method. It seems bolted on.
I'd be +1 on adding a distinct limit() method, and keeping the legacy
list method for masochists who actually enjoy coding in that way.
Adrian
--
Adrian Holovaty
holovaty.com | djangoproject.com
I guess I'm curious as to what's difficult or error-prone about it;
I've never run into a problem where slicing was the cuplrit, and it
feels like a more intuitive and Pythonic syntax than the "limit" and
"offset" arguments we used to have in the pre-magic-removal days.
> I think one of the reasons it seems magic is that
> it's one of the *rare* cases in which we use a Python
> "magic-syntax-ism" (like operator overloading, for instance) in the
> framework, plus the fact that, what, *every* other piece of QuerySet
> functionality uses a method. It seems bolted on.
In a way this feels like the same sort of debate as the one over
whether to implement __nonzero__() as a way of checking for an empty
QuerySet, or to use a dedicated exists() method; going one way is more
"Pythonic", going the other way is more "SQL-ish". I'd prefer to stick
to one way to do things, and probably it'd be a good idea to have a
design discussion about which way Django will lean on these issues.
--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."
And.. it's late and I'm possibly not thinking clearly, but I feel like
fleshing out what I've already said, because this is something I have
a strong opinion on.
Let's step back from the case at hand and consider something else so
it'll be a bit more objective to look at. That means another "magic"
Python method, so I'll pick the __str__()/__unicode__() combo we do on
model classes.
To someone who's not familiar with Python, the fact that defining
these methods will cause a complex object to -- in certain
circumstances -- "magically" become a string is undoubtedly confusing.
But the solution to that isn't to stop using them, it's to help people
become familiar with the conventions of Python programming; if we
switched to, say, defining 'toUnicodeString()' and 'toByteString()' on
models, a lot more people would be able to immediately pick up Django
and grok what's going on, but we'd be going against the grain of
Python, which provides __unicode__() and __str__() for these purposes
already, and we'd be confusing people who are familiar with Python and
who expect that "convert Object X to a string" is going to call
__unicode__() or __str__().
I don't think there's anybody here who'd advocate adding explicit
string-generating methods on models just for the sake of letting
people avoid the need to understand how Python works; we'd be working
against the language in a painful way, and in the long run it'd hurt a
lot more than it helped.
Getting back to the issue at hand, slicing a QuerySet is the same
situation: a lot of people would undoubtedly grok things more quickly
if there were explicit "limit()", "offset()", etc. methods on
QuerySet, but Python already provides a standard language feature for
objects which want to emulate a sequence and allow users to retrieve
only a portion of the sequence, and defining our own separate set of
methods for this would be working against the language in a big way,
and we'd be doing it, essentially, so that people don't have to learn
how Python works. Which is a disservice to everyone who uses Django:
1. Experienced Python programmers would wonder why the hell we're not
just using the features Python provides for this use case.
2. Inexperienced Python programmers would just get bitten the first
time they tried to slice some other sequence type and be right back in
the same boat they were in originally.
And, ultimately, referring to a standard language feature as "magic"
does a large disservice both to Python and to people who are trying to
learn Django and Python: the slicing syntax, and the way it's
implemented, is no more or less "magical" than, say, a Java class
which implements standard interfaces out of java.util or java.lang.
So if it's possible to have a negative vote which involves aleph
numbers or other such suitably infinite quantities, that's how I'd
vote on this change.
I suppose it depends on what you mean by "conforms"; if we wanted to
run with that, I'd ask why you can iterate directly over a QuerySet in
a for loop instead of doing something like, say
queryset = MyModel.objects.all()
for obj in queryset.iterate():
print obj
After all, iteration is implemented using a special Python feature
just the same as slicing, and it's not necessarily obvious that a
QuerySet is something you ought to be able to iterate over without
calling a special method.
> I do think you put it aptly in asking do we want to be more SQL-style or
> Python-style. Ultimately it boils down to people understanding the framework
> better, and keeping it as is may infact do that. I think it is something
> that needs to be considered quite carefully. Promoting slicing as the
> recommended way, but still allowing a limit() function in there to conform
> with the rest of the db api. Conformity, to a set standard (which in this
> case doesn't seem to exist - which is another reason I am +1 on the DEP
> idea), is better than doing it just cause you can.
I'm very much against maintaining two separate ways to do the same
thing. And I'm very strongly in favor of using the "Pythonic" solution
as opposed to the "SQL" solution.
> What the issue was (and the reason I bought this up) was that it was
> understood that the python functionality was executed after the method was,
> and therefor there was an understanding that it wouldn't make any
> difference:
That's the point, though, where it's important to know how Python's
slicing syntax is applied to custom classes, and to go look to see how
the slicing is implemented. We could do a better job of documenting
exactly what slicing does (since it translates into LIMIT/OFFSET in
the DB query), but that still doesn't strike me as an argument for
killing off what is, essentially, the standard Python syntax for this
sort of operation.
> Its something that has been solved by marking with a comment, however I'm
> firmly a fence sitter at this point, so I'm ready to be swung either way.
> But I don't think your comments so far have done so.
After I wrote my second reply to Adrian, I decided to really go
all-out and just write a blog entry about this; my main beef is with
calling things "magic" when really they're just standard Python
features, but maybe it'll help shed some light on the larger
discussion:
http://www.b-list.org/weblog/2007/dec/03/magic/
Adding limit() -- which will also require an offset() method -- is just
as error prone and is less intuitive than the current system (which
conforms to normal Python behaviour). All over the world conversations
such as the following will occur:
Jim:
I want to select the third, fourth and fifth items from
this queryset. What's the best way to do that? I always
forget the syntax.
Jane:
I think you limit() your queryset to five items and then
offset() to the third and it will work. You could also
offset() first to item three and then limit() to three.
Jim:
Really, don't limit() and offset() commute?
Jane:
Well, all the other queryset methods apply
incrementally, so these two should as well, shouldn't
they? In the latter case you first apply the offset,
which restricts the result set and *then* limit it
further. offset(3).limit(5) would give you items three
through seven, wouldn't it?
Jim:
But it's deliberately written as a leaky abstraction of
the SQL extensions called limit and offset. So why
doesn't it behave the same as they do? Maybe that's the
way it works. We're going to have to look it up
again. :-(
I guess it's not possible to have both, so the Django
developers just chose arbitrarily. That sucks... it's
impossible to use logic to understand this. They should
be called chop_first() and interval() then.
Jane:
Yeah, what we really need is a way to specify both at
once. A method that takes the lower and upper bound.
That would remove the ambiguity. Maybe you could leave
out one or the other. We could call it ... *thinks for a
bit* ... __getitem__()!
Jim:
Agreed. We're really taking a subsection of a sequence
here, right? Leaky abstractions aside, Django is
fundamentally presenting a Python view of the data and
only using the database as a persistence mechanism.
Slicing is the normal way to extract sections from
sequences in Python. Wonder why they didn't do that?
Ancient John (older programmer, who's been around the company
for years):
They used to do that. Then Google cut off the oxygen to
the django-developers group and people stopped thinking
clearly. Confusion spread throughout the land.
Apparently the current method is deemed superior because
it keeps people talking to each other and doesn't let
familiarity and standard patterns settle in. It's called
social engineering: you have to be more social to get
stuff done.
This is all very disappointing.
Malcolm
> Ancient John (older programmer, who's been around the company
> for years):
> They used to do that. Then Google cut off the oxygen
> to the django-developers group and people stopped thinking clearly.
> Confusion spread throughout the land. Apparently the current method
> is deemed superior because it keeps people talking to each other and
> doesn't let familiarity and standard patterns settle in. It's called
> social engineering: you have to be more social to get stuff done.
Lol, very good!
Luke
--
"Mistakes: It could be that the purpose of your life is only to serve
as a warning to others." (despair.com)
Luke Plant || http://lukeplant.me.uk/
Here's a specific problem that's bitten me a number of times. I want
to get a list of model objects and do something special with the first
one. I want to pass both the full list and the first one to the
template.
objects = MyModel.objects.filter(site=1)
first_one = objects[0]
do_something_special(first_one)
return render_to_response('t.html', {'objects': objects, 'first':
first_one})
Although this works, it performs two database queries -- one to
retrieve all of the objects, and one to retrieve the first one (with a
"LIMIT" clause). That's unacceptable.
The (unintuitive) way to fix the problem is to wrap the first line in list() --
objects = list(MyModel.objects.filter(site=1))
The counter argument here is, "Well, that's exactly what you have to
do with any iteratable." And my counter argument to that is, "Yeah,
but normal iterables raise TypeError when you try to slice them."
The confusion here stems from the fact that a QuerySet *sometimes*
behaves like a list, but not always. After the QuerySet has been
evaluated, then the slicing syntax simply selects items from the list,
but before the QuerySet has been evaluated, the slicing syntax acts as
a LIMIT/OFFSET. Different things happen depending on the context --
and that's what I find confusing.
> Let's step back from the case at hand and consider something else so
> it'll be a bit more objective to look at. That means another "magic"
> Python method, so I'll pick the __str__()/__unicode__() combo we do on
> model classes.
Sorry for leading you astray and apparently causing you to spend a
long time writing a blog entry about what is and isn't magic! I
shouldn't have used the word "magic," as it's a loaded term and has
different meanings in different contexts (much like QuerySet!). I'll
restate my philosophical problem with this API without using that
word. :-)
My problem with the slicing syntax is that it's the *only* part of the
QuerySet API that uses Python syntax hacking -- as in, "the plus sign
now does something different," or "the slice operator now does
something different." Everything else in the API uses normal Python
methods. It's this inconsistency that bothers me.
I'm still +1 on fixing this wart, but it won't be the end of the world
if nobody agrees with me. I just want folks to understand my
reasoning.
I may be alone in this, but the slicing syntax actually seems a bit
non-Pythonic to me. Query sets are conceptually similar in many ways
to iterators, yet iterators (at least the ones in the standard
library) are not slicable. In order to do that, you have to use
itertools.islice, a function call. Similarly, to reverse an iterator,
you don't slice it with [::-1]; you call reversed, another function.
It seems to me the difference is that slicing, like attribute lookup,
is intuitively a simple operation that works on data already in
memory. If the operation is going to do something substantially more
complex than that, as QuerySet does, then implementing it using
__getitem__ or a property is just misleading; it's actually a method,
and it ought to be denoted as such.
In short, the fact that it uses a feature of Python doesn't
necessarily make it Pythonic in my view. Count me as +0 on the
change.
-root
But what would the situation be with a new limit() method...
objects = MyModel.objects.filter(site=1)
first_one = objects.limit(1)
do_something_special(first_one)
return render_to_response('t.html', {'objects': objects, 'first': first_one})
If limit() returns a new QuerySet, you're still going to have two queries.
Now, maybe we could introduce some mechanics into QuerySet so that your new
QuerySet returned by limit() would keep a reference to the original QuerySet
to use for slicing it's cached results (evaluating it if it hasn't already
been evaluated). Of course, something like this would also fix the two
queries with slicing :)
On a similar note, we have two-query situations happening with count() and
values() too...
objects = MyModel.objects.filter(site=1)
count = objects.count()
objects = MyModel.objects.filter(site=1)
skinny_objects = objects.values('field1')
both of which could use an optimization if their original QuerySet has already
been evaluated.
> The (unintuitive) way to fix the problem is to wrap the first line in list() --
>
> objects = list(MyModel.objects.filter(site=1))
Another option might be to make slicing return a list and not a QuerySet,
which is already happening if you use a step with your slice.
> The counter argument here is, "Well, that's exactly what you have to
> do with any iteratable." And my counter argument to that is, "Yeah,
> but normal iterables raise TypeError when you try to slice them."
And I say QuerySets are more like lazy-evaluated lists. lists are iterable
and can be sliced.
> The confusion here stems from the fact that a QuerySet *sometimes*
> behaves like a list, but not always. After the QuerySet has been
> evaluated, then the slicing syntax simply selects items from the list,
> but before the QuerySet has been evaluated, the slicing syntax acts as
> a LIMIT/OFFSET. Different things happen depending on the context --
> and that's what I find confusing.
I don't see how this fact would lead to confusion as this is only how things
work behind the scenes. To the developer using a QuerySet, it behaves like a
list no matter if the QuerySet has been evaluated or not. I do agree however,
that we be efficient as possible on the backend.
> My problem with the slicing syntax is that it's the *only* part of the
> QuerySet API that uses Python syntax hacking -- as in, "the plus sign
> now does something different," or "the slice operator now does
> something different." Everything else in the API uses normal Python
> methods. It's this inconsistency that bothers me.
>
> I'm still +1 on fixing this wart, but it won't be the end of the world
> if nobody agrees with me. I just want folks to understand my
> reasoning.
I wouldn't be against adding limit() and offset() to _complement_ the existing
slicing capability. So call me +0 for adding limit() and offset() and -1 for
removing slicing support.
Gary
Providing limit() would be consistent to the rest of the querying API,
since all
query building is done using function calls!
my 2 cents
wolfram
--
cu
Wolfram
> And I say QuerySets are more like lazy-evaluated lists. lists are iterable
> and can be sliced.
I think this hits the point.
What do you think about this: a QuerySet fetches a chunk of elements, say
100, as soon as you request at least one? So aQuerySet[0] could fetch the
first 100 or so elements behind the scene and return the first one, and the
next access would not have to go to the database.
Michael
--
noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg -
Tel +49-911-9352-0 - Fax +49-911-9352-100
http://www.noris.de - The IT-Outsourcing Company
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk -
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689
-1, if the query is a more complex and you even hit an unoptimized query
this will have an effect on the overall performance, since every request
does this!
wolfram
> Michael
>
> --
> noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg -
> Tel +49-911-9352-0 - Fax +49-911-9352-100
> http://www.noris.de - The IT-Outsourcing Company
>
> Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk -
> Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689
>
>
> >
>
--
cu
Wolfram
this causes an IndexError
Model.objects.filter(something)[0]
you have to add the colon before the 0
Model.objects.filter(something)[:0]
the limit(0) would not bring this error, it would not let
first-time-users run into this problem
another 2 cents
wolfram
--
cu
Wolfram
I'm not sure why [:0], would ever be useful -- it always returns an
empty list, right? Did you mean [:1] ? But anyway, the behaviour here
is exactly consistent with python lists:
>>> [][0]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list index out of range
>>> [][:0]
[]
>>> [][:1]
[]
We are not helping newbies by making slicing for QuerySets work
differently -- in fact we've gone to some length to make sure it
behaves as analogously as possible -- it throws the same exceptions
etc.
Luke
--
"My capacity for happiness you could fit into a matchbox without taking
out the matches first." (Marvin the paranoid android)
Luke Plant || http://lukeplant.me.uk/
thx
wolfram
--
cu
Wolfram
No -- I guess I didn't explain myself well enough. In this case, I
wouldn't use limit(). I have two goals:
* Retrieve all objects in the table.
* Do something special with the first one (once the whole list has
been retrieved).
The ideal API would look like this, and it would only run a single query:
objects = MyModel.objects.filter(site=1)
first_one = objects[0]
Adrian
Maybe introduce a resolve() or results() call that just returns list(self)?
--
Patryk Zawadzki
PLD Linux Distribution
Isn't that sort of wanting it both ways -- wanting querysets to be both lazy and
not? Surely if the caller wants the objects fetched *now*, he should have to
indicate that, right? Is this not a reasonable way to do that?
objects = list(MyModel.objects.filter(site=1))
first_one = objects[0]
Maybe another method would make that read nicer:
objects = MyModel.objects.filter(site=1).fetch()
first_one = objects[0]
-Forest
--
Forest Bond
http://www.alittletooquiet.net
Replying to self.
On Tue, Dec 04, 2007 at 11:25:42AM -0500, Forest Bond wrote:
> objects = MyModel.objects.filter(site=1).fetch()
> first_one = objects[0]
Incidentally, qs.fetch() need not be equivalent to list(qs). It could just
trigger qs to cache the results of the query and return qs itself. That would
ensure that the query is only executed once, unless further manipulations are
attempted.
Not that that is necessarily the right thing to do.
Ah, my fault. You didn't mention limit() in your post, I was just reading
through from your post a couple back in the thread.
> * Retrieve all objects in the table.
> * Do something special with the first one (once the whole list has
> been retrieved).
>
> The ideal API would look like this, and it would only run a single query:
>
> objects = MyModel.objects.filter(site=1)
> first_one = objects[0]
Any particular ideas about the implementation?
1) Change slices to always return lists and introduce limit() and offset()
methods that return QuerySets.
2) objects[0] evaluates MyModel.objects.filter(site=1) and then returns the
first item.
3) Try to be smart and do a SELECT * LIMIT 1 if first_one is used first, then
run SELECT * when objects is used. Or, if objects is used first run SELECT *,
then when first_one is used it just take the first item of the result cache of
the QuerySet it cloned.
others ideas?
Gary