TimeDivision.objects.filter(when__ge=datetime(2007,5,14))
Whoops. "Greater or equal" is spelled 'gte' and not 'ge' in Django's
model API. After having fallen into this trap and getting a cryptic
error message (UnboundLocalError: local variable 'new_opts' referenced
before assignment ), I realized that it was finally time for a
suggestion I've been meaning to make:
Use natural operators for filtering, excluding, and attribute
access.
Using named parameter syntax leads to all those awful double
underscores and the potential for hard-to-diagnose spelling errors.
Here's how I'd rewrite some of the examples at
http://www.djangoproject.com/documentation/db_api/#when-querysets-are-evaluated
given support for my preferred syntax:
-------
# use something else if you don't like _
from django.models import field as _
Entry.objects.filter(_.pub_date.year==2006)
-------
from django.models import field as _
Entry.objects.filter(
_.headline.startswith('What')).exclude(
_.pub_date>=datetime.now()).filter(
_.pub_date>=datetime(2005, 1, 1))
-------
# I'm going to stop repeating this now
from django.models import field as _
q1 = Entry.objects.filter(_.headline.startswith('What'))
q2 = q1.exclude(_.pub_date >= datetime.now())
q3 = q1.filter(_.pub_date >= datetime.now())
-------
Entry.objects.exclude(_.pub_date > datetime.date(2005, 1, 3)).exclude(_.headline=='Hello')
-------
Entry.objects.filter(_.pub_date.year==2005).order_by(-_.pub_date, _.headline)
-------
IMO this syntax is:
* less arcane
* much less error-prone
* much easier to write good error messages
How about it?
--
Dave Abrahams
Boost Consulting
www.boost-consulting.com
> -------
> from django.models import field as _
>
> Entry.objects.filter(
> _.headline.startswith('What')).exclude(
> _.pub_date>=datetime.now()).filter(
> _.pub_date>=datetime(2005, 1, 1))
Should be:
Entry.objects.filter(
_.headline.startswith('What') &
~(_.pub_date>=datetime.now()) &
_.pub_date>=datetime(2005, 1, 1))
> -------
> Entry.objects.exclude(_.pub_date > datetime.date(2005, 1, 3)).exclude(_.headline=='Hello')
Should be
Entry.objects.exclude(_.pub_date > datetime.date(2005, 1, 3) | _.headline=='Hello')
> IMO this syntax is:
>
> * less arcane
> * much less error-prone
> * much easier to write good error messages
Did I mention "terser," too?
> How about it?
Some immediate problems:
- Your syntax works for < > <= >= and ==; but what about all the other
Django query operators? How do you come up with a clean representation
for startswith? istartswith? range? month?
- I'm not a huge fan of the _. syntax. Leaving aside the clash with
the existing usage of _ as a shortcut for the i18n gettext function, I
don't see how the degenerate case of filter(name=value) is improved by
making it filter(_.name=value)?
- We've had this discussion before, about a year ago, when we were
doing the magic removal branch. Check the django-dev archives. There
are a lot of blog posts around that time talking about 'reinventing
SQL'. The consensus at the end was that there is a lot of Django code
out there that is using the existing syntax, and the existing syntax
works, so there isn't a great need to change it.
- We are just starting a v1.0 push, so massively changing the query
syntax really isn't on the cards.
In short - thanks for your contribution, but I don't think you're
going to get much traction with this idea.
However, you have obviously been bitten by a bad error message - I
would suggest that a good course of action would be to raise a ticket
with your specific example so that the error reporting of that part of
the query engine can be fixed up.
Yours,
Russ Magee %-)
Maybe I'm missing something, but I don't see how this can work at all.
When Python sees "_.foo >= 6", it is going to evaluate it at runtime
(before calling the function). However, we need the SQL construction
code to see the ">=" bit so that we can pass the right operator through
to the database.
So I believe there is a bigger problem here: Python's syntax parser
won't allow us to get the right information.
Using the "foo__gte = 6" syntax, we get the whole parameter name and can
turn the __gte portion into the right SQL operator.
Regards,
Malcolm
> On 2/10/07, David Abrahams <da...@boost-consulting.com> wrote:
>>
>> Use natural operators for filtering, excluding, and attribute
>> access.
>
>> How about it?
>
> Some immediate problems:
>
> - Your syntax works for < > <= >= and ==; but what about all the other
> Django query operators? How do you come up with a clean representation
> for startswith?
Easy. I showed at least one such example already, which I refine below:
Entry.objects.filter(
_.headline.startswith('What')
^^^^^^^^^^^^^^^^^^^
, _.pub_date<datetime.now()
, _.pub_date>=datetime(2005, 1, 1))
> istartswith?
Entry.objects.filter(_.headline.istartswith=='will')
> range?
Entry.objects.filter(_.pub_date.range(start_date, end_date))
or
Entry.objects.filter(_.pub_date.range==(start_date, end_date))
or
Entry.objects.filter(_.pub_date >= start_date & _.pub_date < end_date)
Yes, because evaluation is lazy the latter can be made just as
efficient as if 'range' were used explicitly.
> month?
Entry.objects.filter(_.pub_date.month==12)
> - I'm not a huge fan of the _. syntax.
Read my examples again:
# use something else if you don't like _
from django.models import field as _
^^^^^^^^^^^^^^^^^
'_' is not imposed by the library; if you don't like it, just use the
library supplied object 'field', or any other identifier you find
appropriate:
Entry.objects.filter(field.pub_date.month==12)
^^^^^
> Leaving aside the clash with the existing usage of _ as a shortcut
> for the i18n gettext function,
It's not a clash; that's what namespaces are for. I sure hope that no
one module could reserve a one-character name for use in all other
modules. Certainly '_' is too useful in various contexts to reserve
it for i18n.
If you're really determined to have both available as globals in an
entire module, you could make your own '_' that accomodates both
functions, since I'm not using the _(...) syntax.
> I don't see how the degenerate case
> of filter(name=value) is improved by making it filter(_.name=value)?
Actually
filter(_.name==value)
^^
First of all, the syntax leaves room for backward compatibility, so if
you want to keep writing
filter(name=value)
you can. The big advantage to my suggestion for the really simple
cases is just that it's consistent with a syntax that makes sense even
for the more complicated ones. It's also more consistent with Python,
where '=' is not an equality operator.
> - We've had this discussion before, about a year ago, when we were
> doing the magic removal branch. Check the django-dev archives. There
> are a lot of blog posts around that time talking about 'reinventing
> SQL'. The consensus at the end was that there is a lot of Django code
> out there that is using the existing syntax, and
Like I said, you can preserve backward compatibility.
> the existing syntax works, so there isn't a great need to change it.
It didn't, and doesn't, work for me. Let me repeat and clarify the
reasons to add this extension:
* less arcane
* much less error-prone, because
- python constrains the syntax to something reasonable
- the syntax for the query uses familiar Python operators in
familiar ways
* much easier give good error messages when the user does make a
mistake (because you don't have to parse a misspelled string
and guess what the user really meant).
* terser for all but the most trivial queries
* much more expressive
From my point of view the gain in expressiveness is 100% worth it and
if I have to do any more serious model manipulation with Django and
it's not implemented yet, I'll do it myself. Expressive interfaces
lead to code that is more often correct the first time and easier to
read and maintain.
Let me also point out that if "the existing syntax works so there
isn't a great need" had been followed from the beginning, Django
wouldn't exist. The core benefit of Django is that it is easy to
express your intentions simply and clearly. To the degree that's
true, it's possible to develop sites quickly. In other words, rapid
development and all the other benefits of django follow from
expressiveness.
> - We are just starting a v1.0 push, so massively changing the query
> syntax really isn't on the cards.
OK. Well, it's easy enough to build a non-intrusive extension that
allows:
objects(Entry).filter(_.pub_date.month==12)
which is nearly as expressive. However, it is not as easy for such an
extension to give good error messages, so there are reasons to build
the functionality into the django core.
> In short - thanks for your contribution, but I don't think you're
> going to get much traction with this idea.
:) I've heard (and overcome) that before
> However, you have obviously been bitten by a bad error message - I
> would suggest that a good course of action would be to raise a ticket
> with your specific example so that the error reporting of that part of
> the query engine can be fixed up.
Done.
> On Sat, 2007-02-10 at 08:30 +0800, Russell Keith-Magee wrote:
>> On 2/10/07, David Abrahams <da...@boost-consulting.com> wrote:
>> >
>> > Use natural operators for filtering, excluding, and attribute
>> > access.
>>
>> > How about it?
>>
>> Some immediate problems:
>>
>> - Your syntax works for < > <= >= and ==; but what about all the other
>> Django query operators? How do you come up with a clean representation
>> for startswith? istartswith? range? month?
>
> Maybe I'm missing something, but I don't see how this can work at all.
> When Python sees "_.foo >= 6", it is going to evaluate it at runtime
> (before calling the function).
Yes. _.foo >= 6 returns an expression tree, an object that represents
the comparison of the foo field with 6. This technique is well-known
among advanced C++ programmers but has also been seen in the Python
world (http://thread.gmane.org/gmane.comp.python.numeric.general/4081,
although I've heard of other instances). Here's enough of a sketch to
represent the above expression. Obviously you'd need more along the
same lines to be able to handle everything:
# represents unary or binary operations
class Node(object):
def __init__(self, op, lhs, rhs=None):
self.op = op # Label used to identify the operation
self.lhs = lhs
self.rhs = rhs
def __getattr__(self, name):
return Node('attr',self,name)
def __ge__(self, other):
return Node('>=',self,other)
field = Node(None,None) # the leaf node
now with _=field,
_.foo >= 6
is the same as
Node('>=', Node('attr', Node(None,None), 'foo'), 6)
which stores all the information needed to construct the correct SQL.
I actually think the former to be clearer than the latter.
--
Lawrence, oluyede.org - neropercaso.it
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair
>> > -------
>> > from django.models import field as _
>> >
>> > Entry.objects.filter(
>> > _.headline.startswith('What')).exclude(
>> > _.pub_date>=datetime.now()).filter(
>> > _.pub_date>=datetime(2005, 1, 1))
>>
>> Should be:
>>
>> Entry.objects.filter(
>> _.headline.startswith('What') &
>> ~(_.pub_date>=datetime.now()) &
>> _.pub_date>=datetime(2005, 1, 1))
>
> I actually think the former to be clearer than the latter.
I'd be happy with the former, especially as the latter is difficult
(though possible) to implement due to the relative precedence of >=
and &.
Frankly, I'd prefer the simpler:
Entry.objects.filter(
_.headline.startswith('What'),
_.pub_date<datetime.now(),
_.pub_date>=datetime(2005, 1, 1))
to either of them, though.
OK, I'd hadn't made the leap that that was why you were introducing the
initial '_' and using attribute access. I understand the concept -- just
the right neurons weren't connected. Thanks for the clarification.
I'm -1 on this for the reasons Russell mentioned and because it adds
extra complexity to the queries (you have this bonus "_." everywhere).
Plus we can't use _ because it's already used by Python's shell and by
our i18n framework (which is a small problem in itself). Three uses for
the same variable is right out.
Regards,
Malcolm
> On Fri, 2007-02-09 at 21:14 -0500, David Abrahams wrote:
>> _.foo >= 6 returns an expression tree, an object that represents
>> the comparison of the foo field with 6. This technique is well-known
>> among advanced C++ programmers but has also been seen in the Python
>> world (http://thread.gmane.org/gmane.comp.python.numeric.general/4081,
>> although I've heard of other instances). Here's enough of a sketch to
>> represent the above expression. Obviously you'd need more along the
>> same lines to be able to handle everything:
>
> OK, I'd hadn't made the leap that that was why you were introducing the
> initial '_' and using attribute access. I understand the concept -- just
> the right neurons weren't connected. Thanks for the clarification.
Yerwelcome.
> I'm -1 on this for the reasons Russell mentioned and because it adds
> extra complexity to the queries (you have this bonus "_."
> everywhere).
From my POV, '__' is a "bonus" and 'gte' is definitely extra
complexity. I have to remember this unholy syntax and try to
understand how it maps onto logic (with this mapping not even
_completely_ defined anywhere except code AFAIK). If that isn't
complexity, I don't know what is. IMO, if you think foo__bar__gte=6
is not complex (especially with the meaning it has) then you've been
using that syntax too long to understand how it looks to a newcomer.
Sorry, I just don't see how this adds any complexity for the user
(except maybe for the very simplest queries, if 3 seems like a large
number and you're counting characters).
> Plus we can't use _
I repeat: I wasn't suggesting the framework supply '_'. I was
suggesting that the framework supply 'field'. In the context where
you want to do queries, you can
from ... import field as _
or, for any xxx you like,
from ... import field as xxx
or just,
from ... import field
and it would still be a vastly more-expressive interface.
> because it's already used by Python's shell and by our i18n
> framework (which is a small problem in itself). Three uses for the
> same variable is right out.
I'd be really, really surprised if there weren't other variable names
that are re-used in three different contexts. :)
I'm also not a fan of the fact that, while claiming to make the
end-user syntax less arcane, it would make the under-the-hood
implementation *considerably* more arcane.
On 2/9/07, David Abrahams <da...@boost-consulting.com> wrote:
> Like I said, you can preserve backward compatibility.
I'm also not sure this point came across quite as well as it should
have: we did a massive ground-up rewrite of most of this stuff last
year, and there were major, in-depth discussions about it. The lookup
syntax we have now as a result of that is the lookup syntax we've
promised to keep until Django 1.0, so unless someone demonstrates a
literally earth-shaking enhancement, I think it's best to keep this
API stable and concentrate on all the other bits which need work
before the 1.0 release.
> It didn't, and doesn't, work for me. Let me repeat and clarify the
> reasons to add this extension:
We did read it the first time ;)
Unfortunately, no API will ever work for everyone -- it's the old "you
can't please all of the people all of the time" problem. Sooner or
later we have to accept that and leave well enough alone -- the DB
lookup syntax doesn't seem to generate a whole lot of complaints or
strange errors (as opposed to other things which most certainly do),
so again I don't think there's enough justification to divert the
necessary development resources into building yet another query layer
when we've already got one that's guaranteed stable until 1.0
> From my point of view the gain in expressiveness is 100% worth it and
> if I have to do any more serious model manipulation with Django and
> it's not implemented yet, I'll do it myself. Expressive interfaces
> lead to code that is more often correct the first time and easier to
> read and maintain.
I'm not sure how typing "<=" instead of "lte" is inherently more
expressive... the self-declared most expressive language on the planet
uses 'eq' as an equality operator, after all ;)
And again, while not trying to bash on this, I think all you've
demonstrated is a subjective gain for your use, and unfortunately
that's not enough to re-open a stable API and start rewriting it all
again -- this subjective gain for you would come at a high cost to
Django in terms of the time and effort which would have to be directed
into building out this new API, testing it, verifying it doesn't break
anything, etc.
> which is nearly as expressive. However, it is not as easy for such an
> extension to give good error messages, so there are reasons to build
> the functionality into the django core.
Getting good error messages is a goal I'd be happy to help out with,
but we don't need to rewrite the entire query system to do that -- the
one we have right now is perfectly amenable to better error messages
for end users. So again, I don't see enough of a gain to justify
re-opening the database API.
--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."
> First thoughts: subjectively, I'm not sure it'd be any prettier than
> what we have now.
Can't argue with your subjective opinion. But I suggest you ask some
newbies what they think. Also, "prettiness" is not really one of my
goals, though I believe that generally comes along with
expressiveness.
> I'm also wary of overloading the concepts of the standard Python
> operators when whet we're really doing is translating to SQL
> operations -- sooner or later, there's going to be an impedance
> mismatch.
Just like there's an impedance mismatch when Python using mathematical
operators? What it's "really doing" is translating to bytecode
representations of operations on machine internals. ;-)
This sort of overloading makes some people uneasy, I know. I have a
lot of experience with it, so am very familiar with both its benefits
and limitations. It is true that you can't make everything
transparent. For example, Python doesn't let you overload 'and' and
'or', so you need to do something else for those. However, what you
end up doing for 'and and 'or' should be no worse than what you have
now (after all, what you have now will work)
> I'm also not a fan of the fact that, while claiming to make the
> end-user syntax less arcane, it would make the under-the-hood
> implementation *considerably* more arcane.
I *seriously* doubt that. This feature is really simple to implement.
Maybe I'll have to code it to prove that, but from the number of "I'm
not a fan of..." responses I'm seeing here, I doubt that knowing it's
simple would make much difference to any of the people who've
responded.
> On 2/9/07, David Abrahams <da...@boost-consulting.com> wrote:
>> Like I said, you can preserve backward compatibility.
>
> I'm also not sure this point came across quite as well as it should
> have: we did a massive ground-up rewrite of most of this stuff last
> year, and there were major, in-depth discussions about it.
I understand that.
> The lookup syntax we have now as a result of that is the lookup
> syntax we've promised to keep until Django 1.0, so unless someone
> demonstrates a literally earth-shaking enhancement, I think it's
> best to keep this API stable and concentrate on all the other bits
> which need work before the 1.0 release.
I can appreciate that; I certainly didn't intend to suggest this was a
"must have before 1.0" feature.
[I'm responsible for getting a "must have before 1.0" feature into an
ISO standard when the standard was already late and supposed to be
closed to large changes, which never could have happened without
appreciating the pressure the committee was under already. So believe
me, I understand just what you are going through. It's not a struggle
I care to repeat, either.]
>> It didn't, and doesn't, work for me. Let me repeat and clarify the
>> reasons to add this extension:
>
> We did read it the first time ;)
I wasn't just repeating myself. I added some material I thought would
help people to better understand my position.
> Unfortunately, no API will ever work for everyone -- it's the old "you
> can't please all of the people all of the time" problem.
Of course not. But not all qualities of an API are matters of
personal preference or opinion.
> Sooner or later we have to accept that and leave well enough alone
> -- the DB lookup syntax doesn't seem to generate a whole lot of
> complaints or strange errors (as opposed to other things which most
> certainly do), so again I don't think there's enough justification
> to divert the necessary development resources into building yet
> another query layer when we've already got one that's guaranteed
> stable until 1.0
If that's the case, I agree.
>> From my point of view the gain in expressiveness is 100% worth it and
>> if I have to do any more serious model manipulation with Django and
>> it's not implemented yet, I'll do it myself. Expressive interfaces
>> lead to code that is more often correct the first time and easier to
>> read and maintain.
>
> I'm not sure how typing "<=" instead of "lte" is inherently more
> expressive...
Simple: it uses an existing notation, in the usual way, for an concept
with which the user is already familiar.
What you have done with 'xxx__yyy__gte=zzz' is invent a new
domain-specific language for expressing predicates on Python objects
(that's the high-level view that Django mostly appears to trying to
supply) or, if you insist, doing database queries (that's the
low-level view).
The closer a domain-specific syntax comes to directly representing the
concepts the user is manipulating in ways already familiar to the
user, the more expressive it is. Both the high-level syntax and the
low-level syntax use '>=' and not 'gte' to represent the concept, so I
don't see any grounds for a claim that 'gte' is as expressive.
> the self-declared most expressive language on the planet
> uses 'eq' as an equality operator, after all ;)
I suppose you mean lisp? Yeah, I never understood that claim. Too
much syntactic regularity hurts expressiveness.
> And again, while not trying to bash on this, I think all you've
> demonstrated is a subjective gain for your use,
That depends on the criteria used by this group to judge the quality
of an API. I have laid out some objective criteria for judging the
quality of a proposed API. I understand your criteria for judging the
feasibility of immediately adopting my proposal, but not your criteria
for judging its quality.
> and unfortunately that's not enough to re-open a stable API and
> start rewriting it all again -- this subjective gain for you would
> come at a high cost to Django in terms of the time and effort which
> would have to be directed into building out this new API, testing
> it, verifying it doesn't break anything, etc.
Sure; I guess I'll build my layer on top of django in that case.
Trying to take the temperature here: is there any sympathy here for
the idea of adding it to contrib/, or is the whole idea simply
anathema to this group?
David,
Looks like you don't like Django's database API. Sorry you don't like
it, but we can't please everybody. At this point, as other people have
explained, we're open to making gradual changes, but not fundamental
changes like the one you're proposing.
I think you'd enjoy using Dejavu, a Python database library that uses
Python expressions to specify "WHERE" clauses.
http://cheeseshop.python.org/pypi/Dejavu/1.5.0RC1
The nice thing about Django is that you can still use the framework
(with a couple of limitations) using another model layer, such as
Dejavu.
Adrian
--
Adrian Holovaty
holovaty.com | djangoproject.com
However, you have obviously been bitten by a bad error message - I
would suggest that a good course of action would be to raise a ticket
with your specific example so that the error reporting of that part of
the query engine can be fixed up.
> On 2/9/07, Russell Keith-Magee <freakb...@gmail.com> wrote:
>
> However, you have obviously been bitten by a bad error message - I
> would suggest that a good course of action would be to raise a ticket
> with your specific example so that the error reporting of that part of
> the query engine can be fixed up.
>
> The fix for #2348 changes the error from "Unbound local error" to "Type error: cannot resolve
> keyword [whatever] into field".
Illustrating just why the current syntax is hard to give good error
messages for ;-)
> On 2/9/07, David Abrahams <da...@boost-consulting.com> wrote:
>> Entry.objects.filter(
>> _.headline.startswith('What') &
>> ~(_.pub_date>=datetime.now()) &
>> _.pub_date>=datetime(2005, 1, 1))
>
> David,
>
> Looks like you don't like Django's database API.
I like some of it, but, well, not the '__' part. But my likes aren't
really the point here.
> Sorry you don't like it, but we can't please everybody. At this
> point, as other people have explained, we're open to making gradual
> changes, but not fundamental changes like the one you're proposing.
It's a pure extension, not a fundamental change... but, whatever.
> I think you'd enjoy using Dejavu, a Python database library that uses
> Python expressions to specify "WHERE" clauses.
>
> http://cheeseshop.python.org/pypi/Dejavu/1.5.0RC1
>
> The nice thing about Django is that you can still use the framework
> (with a couple of limitations) using another model layer, such as
> Dejavu.
Considering I can't even reach Dejavu's home page, I think I'd rather
write my query layer myself.
"Karen Tracey" <kmtr...@gmail.com>
writes:
> The fix for #2348 changes the error from "Unbound local error" to "Type error: cannot resolve
> keyword [whatever] into field".
Illustrating just why the current syntax is hard to give good error
messages for ;-)
> "Karen Tracey" <kmtr...@gmail.com>
> writes:
> > The fix for #2348 changes the error from "Unbound local error" to "Type error: cannot
> resolve
> > keyword [whatever] into field".
>
> Illustrating just why the current syntax is hard to give good error
> messages for ;-)
>
> Well it is a big improvement in that it points you to exactly where it runs into trouble,
> whereas the unbound local error was a confusing red herring.
Yes, it's better. But "cannot resolve keyword [whatever] into field"
is also a red herring when the misspelled bit was never intended to be
a field name in the first place. Pity the poor user when the converse
happens and "gate" is misspelled as "gte"!
> (I have no energy at the moment to consider the pros and cons of
> changing the syntax here; I was just trying to save anyone the
> trouble of opening a ticket for something that had just been fixed.)
Thanks; I didn't mean to seem ungrateful for the report. I just have
bigger fish to fry. ;-)
I can't really claim to be a complete newbie, as I've spent a couple
of days
playing with Django, but I didn't find any problems at all with the
__gte syntax.
It might not be the most intuitive system, but after you've learned
it, it makes
sense. It''s also very easy to learn with the current documentation,
so I fail
to see the problem.
Just my 0.02.
--
Alf
> Trying to take the temperature here: is there any sympathy here for
> the idea of adding it to contrib/, or is the whole idea simply
> anathema to this group?
For me, the idea looks great. I will not hold my breath waiting it to
be implemented in 0.97 or even 1.5, but your way is definitely more
expressive (and I suspect it will be cheaper for the CPU as it gives
more work to the compiler leaving less for the runtime).
Do you plan to develop an extension yourself?