Overloading and? was <RE: Should I prefer an external database>
Bjorn Pettersen
[... + SQLObject example]
> for paper in Paper.select(AND(Paper.q.title == 'foo',
> Paper.q.author.startswith('Bob'))):
Does anyone know the reason for not allowing an overload of the and
operator? (of course, the only use case I've seen is the one above
[thrice, first time in 97] and alternate logical calculi [e.g. fuzzy
logic] -- so not much different from some of the other operators? :-)
-- bjorn
Terry Reedy
operator?
I presume because 'and' and 'or' are already defined for all Python
objects (that have not disabled logic by raising an exception when
asked for truth value).
TJR
Ian Bicking
> > for paper in Paper.select(AND(Paper.q.title == 'foo',
> > Paper.q.author.startswith('Bob'))):
>
> [thrice, first time in 97] and alternate logical calculi [e.g. fuzzy
> logic] -- so not much different from some of the other operators? :-)
Certain operators (and, or, not, is) can't be overloaded. (Actually, up
until 2.1, the kind of overloading I'm doing with the comparison
operators wasn't possible)
For and, or, and not you can overload (I believe the name is)
__nonzero__, and they use that to determine if the value should be
considered true or false. But for an expression generator like this I
need to overload each operator so it returns a specific value, not just
True or False. In this case a == b returns an object that evaluates to
"a = b" (for use in SQL).
Ian
Andrew Dalke
> for paper in Paper.select(AND(Paper.q.title == 'foo',
> Paper.q.author.startswith('Bob'))):
> Does anyone know the reason for not allowing an overload of the and
> operator?
The 'and' and 'or' operators work on the boolean-ness of the objects,
and apply short-circuit behaviour. This is different than other binary
operators, which don't short-circuit.
For example, consider
counter = 0
class Funky:
def __nonzero__(self):
global counter
print "Hello!"
assert counter == 0
counter = 1
return 1
>>> counter = 0
>>> a = Funky() or Funky()
Hello!
>>> a
1
>>>
Note that __nonzero__ was only called once, not twice.
There's no way to implement that behaviour in Python, so
it cannot be overridden. Eg, suppose there is an "__and__".
You might think it could look like
def __and__(self, other):
if bool(self): return self
if bool(other): return other
return False
However, this doesn't work because of the short-circuiting
print 1 and 1/0
prints '1', and never evalutes 1/0.
BTW, what I've done for other projects which build up a parse
tree then evalute it is to hijack the arithmetic operators. I've
used '*' for and, '+', for or, and '-' for 'but not'. In that case, the
above expression is
> for paper in Paper.select(Paper.q.title == 'foo' +
> Paper.q.author.startswith('Bob')):
Andrew
da...@dalkescientific.com
A. Lloyd Flanagan
I'm going to guess it's because a) it's a boolean operation, not an
arithmetic one, and therefore b) overloading it could cause incredible
mischief with the language. Imagine:
class C:
def __and__(self, some_other_thing):
return (self or some_other_thing)
Now suddenly if statements don't work for some objects, all kinds of
internal code probably fails for no apparent reason, etc.
Other reasons would be performance impact on a fundamental language
construct, and possibly that no one could think of a good reason why
you'd want to do that.
Ian Bicking
> BTW, what I've done for other projects which build up a parse
> tree then evalute it is to hijack the arithmetic operators. I've
> used '*' for and, '+', for or, and '-' for 'but not'. In that case, the
> above expression is
>
> > for paper in Paper.select(Paper.q.title == 'foo' +
> > Paper.q.author.startswith('Bob')):
This won't work, though, because of the order of precedence. It parses
like (Paper.q.title == ('foo' + Paper.q.author.startswith('Bob'))).
Better operators are &, |, and ~ (the binary logic operators). You can
use these with SQLObject, but I don't like to, because they have the
same problems of precedence and they are more trouble than they are
worth. AND() isn't pretty, but it's very straight-forward to use and
unlikely to cause surprise.
Ian
Steven Taschuk
> Bjorn Pettersen:
[...]
> > Does anyone know the reason for not allowing an overload of the and
> > operator?
>
> The 'and' and 'or' operators work on the boolean-ness of the objects,
> and apply short-circuit behaviour. This is different than other binary
> operators, which don't short-circuit.
> There's no way to implement that behaviour in Python, so
> it cannot be overridden. Eg, suppose there is an "__and__".
> You might think it could look like
>
> def __and__(self, other):
> if bool(self): return self
> if bool(other): return other
> return False
__and__ could be passed other as a callable which computes and
returns the second operand. Then the standard __and__ could be
def __and__(self, other):
if bool(self):
return self
return other()
(Must return the second object when it is false, thus:
>>> 1 and []
[]
rather than False.)
--
Steven Taschuk stas...@telusplanet.net
"I may be wrong but I'm positive." -- _Friday_, Robert A. Heinlein
Bjorn Pettersen
>
> Bjorn Pettersen:
> > From: Ian Bicking [mailto:ia...@colorstudy.com]
> [... + SQLObject example]
>
> > for paper in Paper.select(AND(Paper.q.title == 'foo',
> > Paper.q.author.startswith('Bob'))):
>
> > operator?
>
> The 'and' and 'or' operators work on the boolean-ness of the objects,
> and apply short-circuit behaviour. This is different than
> other binary
> operators, which don't short-circuit.
Ah! Thanks.
-- bjorn
Andrew Dalke
> __and__ could be passed other as a callable which computes and
> returns the second operand. Then the standard __and__ could be
> def __and__(self, other):
> if bool(self):
> return self
> return other()
So a new idiom and new code (to generate a function from a set of
unevaluated code), just to allow overriding booleans? Seems
rather excessive to me.
Gets pretty complicated with code like
print value or sys._getframe().f_lineno
since the function introduces a new frame.
OTOH, this could lead to an easy 'if/else' expression function.
And generically speaking, adding code blocks like this leads
towards Smalltalk and Ruby.
Andrew
da...@dalkescientific.com
Bjorn Pettersen
> > __and__ could be passed other as a callable which computes and
> > returns the second operand. Then the standard __and__ could be
> > def __and__(self, other):
> > if bool(self):
> > return self
> > return other()
>
> So a new idiom and new code (to generate a function from a set of
> unevaluated code), just to allow overriding booleans? Seems
> rather excessive to me.
It occurred to me that C++ allows overloading of && (non c-s), while the
regular && is c-s.
Could it be that c-s is only useful in domains with exactly two elements
(true, false). I.e. don't worry about c-s for overloaded __and__ since
(a) the class you're overloading it on will probably not be limited to
two instances, and (b) even if it was, how would you know c-s is
desireable or even appropriate?
-- bjorn
Steven Taschuk
> Steven Taschuk
> > __and__ could be passed other as a callable which computes and
> > returns the second operand. Then the standard __and__ could be
> > def __and__(self, other):
> > if bool(self):
> > return self
> > return other()
>
> So a new idiom and new code (to generate a function from a set of
> unevaluated code), just to allow overriding booleans? Seems
> rather excessive to me.
Yes.
I do not (and did not) endorse this notion for __and__. I simply
do not agree with the previous claim that there's no way to
implement overloading and short-circuiting for the same operator.
No *good* way given how Python works now, perhaps.
> Gets pretty complicated with code like
>
> print value or sys._getframe().f_lineno
>
> since the function introduces a new frame.
A nice example.
--
Steven Taschuk stas...@telusplanet.net
Receive them ignorant; dispatch them confused. (Weschler's Teaching Motto)
Andrew Dalke
> I do not (and did not) endorse this notion for __and__. I simply
> do not agree with the previous claim that there's no way to
> implement overloading and short-circuiting for the same operator.
>
> No *good* way given how Python works now, perhaps.
Ahhh, a nit-picker, eh? :)
Let's see. I said:
] There's no way to implement that behaviour in Python, so
] it cannot be overridden.
I submit that a language which worked in your suggested manner
would not be Python.
The nit is now in your court!
:)
Andrew
da...@dalkescientific.com
Steven Taschuk
[...overloadable, short-circuiting operators...]
> Ahhh, a nit-picker, eh? :)
>
> Let's see. I said:
> ] There's no way to implement that behaviour in Python, so
> ] it cannot be overridden.
>
> I submit that a language which worked in your suggested manner
> would not be Python.
>
> The nit is now in your court!
Hm. I'm not sure I understand what would make such a language not
Python.
I agree, of course, that no known Python has such special __and__
and __or__ methods. But that no *known* Python works as I
suggested does not mean that no *possible* Python works thus.
(Consider that not long ago the yield statement was not part of
any known Python. And imho generators are a much larger change to
the language than two new special methods with unusual arguments.)
--
Steven Taschuk Aral: "Confusion to the enemy, boy."
stas...@telusplanet.net Mark: "Turn-about is fair play, sir."
-- _Mirror Dance_, Lois McMaster Bujold
Andrew Dalke
> Hm. I'm not sure I understand what would make such a language not
> Python.
I believe apologies are due. It appears I misread flippancy into
your response, so this one of mine is now more serious.
> I agree, of course, that no known Python has such special __and__
> and __or__ methods. But that no *known* Python works as I
> suggested does not mean that no *possible* Python works thus.
> (Consider that not long ago the yield statement was not part of
> any known Python. And imho generators are a much larger change to
> the language than two new special methods with unusual arguments.)
I don't think generators are that large a change. Pre-generators I
wrote a lot of code which looked like this
class TwoLineIterator:
def __init__(self, infile):
self.infile = infile
self._n = 0
def next(self):
line = self.infile.readline()
if not line:
return None
self._n = self._n + 1
return [line, self.infile.readline()]
def __getitem__(self, i):
if self._n != i:
raise TypeError("forward iteration only!")
x = self.next()
if x is None:
raise IndexError(i)
return x
I got so I could crank out the code almost without thinking.
With iterators, I added
def __iter__(self):
return iter(self, None)
With generators I changed the whole thing to
def TwoLineIterator(infile):
while 1:
line = infile.readline()
if not line:
break
yield [line, infile.readline()]
Externally, the interface is exactly the same
for data in TwoLineIterator(open("input.txt")):
...
Conceptionally then, I see yield (on top of the iterator protocol)
as a great way to simplify building an existing idiom. And
in addition, it lets me turn highly stateful operations (like a
tree or graph traversal) into iterators. When the element
generation is several loops deep, it's even harder. Without
generators, I ended up using threads to get the iterator-like
feel I wanted.
But Python the language did let me get that feel, even in the
1.4 days, and adding generators to Python has not affected
that feel.
Consider though the "two new special methods with
unusual arguments." Assume the details of getting the stack
frames, etc have been worked out, along with problems of
not knowing the __and__ method even exists until the code
is run. (It could be added after the class is created.)
You're left with functions which behave different from any
other function, and where the behaviour cannot be used
anywhere else.
First, they behave different. What's the object passed to
the function? It's callable, it contains some execution stack
information, to get the locals. It's a pretty complicated
thing. Does that data type already exist? Although I don't
know the internals that well, I don't think so, which means
it needs to be written, documented, available for introspection,
etc.
Can that object be saved? Eg, what does this do?
class Spam:
def __and__(self, other):
self.saved = other
return other()
spam = Spam()
try:
a = spam and 1/0:
except ZeroDivisionError:
pass
spam.saved()
For some languages, like Smalltalk and Ruby, this makes sense.
Those languages support code blocks, which can be passed
around and executed. But Python does not.
Were Python to have code blocks, then it should be applied
more generally to Python code. Consider the if/else PEP of
a couple months ago. With code blocks, the following would
be possible
ifelse(a != 0, 1/a, None)
which would pass the 1/a and None as independent code
blocks, to be executed based on the evaluation of a != 0.
And Guido said that this sort of function would not be one
of the ways to resolve the PEP, which I take to mean that
it's not really a Python solution.
BTW, this would also require some sort of prototyping, to
specify which fields are to be passed as blocks vs. evaluated
before passing, as in
def ifself(cond, if_true is code, if_false is code):
if cond:
return if_true()
return if_false()
but that just doesn't feel right, which I take to mean it
isn't really Python. It looks like Python, but it doesn't
feel like it.
And that's why I say that code which handles this sort
of special __and__ and __or__ case, if added to Python,
would produce code which wasn't really Python. It may
be related to Python -- given the feedback I've seen from
people who like code blocks in other languages, I can
understand why people would like them in a Python-like
language. But it would not be Python.
Andrew
da...@dalkescientific.com.
Ian Bicking
> Can that object be saved? Eg, what does this do?
>
> class Spam:
> def __and__(self, other):
> self.saved = other
> return other()
>
> spam = Spam()
> try:
> a = spam and 1/0:
> except ZeroDivisionError:
> pass
> spam.saved()
Arrr matey... there be continuations. Dark waters indeed!
> For some languages, like Smalltalk and Ruby, this makes sense.
> Those languages support code blocks, which can be passed
> around and executed. But Python does not.
I suspect that in Smalltalk you cannot actually overload and:. It is a
quiet optimization that Smalltalks do, that turn particular methods into
special bytecode. Things like ifTrue:ifFalse:, and to:do: are not
compiled to method calls, and I suspect the same is true of and:. It's
just too inefficient.
I don't actually know how Ruby does and...? Syntactically distinct like
Python, or as a method call like Smalltalk? I expect more like
Python... either way I'm sure they all do the same thing deep down.
> Were Python to have code blocks, then it should be applied
> more generally to Python code. Consider the if/else PEP of
> a couple months ago. With code blocks, the following would
> be possible
>
> ifelse(a != 0, 1/a, None)
>
> which would pass the 1/a and None as independent code
> blocks, to be executed based on the evaluation of a != 0.
Of course, ifelse would be best implemented as a macro, not with code
blocks (which presumably would require some syntax, like
ifelse(a != 0, `1/a`, `None`)). Macros are of course compile-time, not
runtime, and Python deliberately does very little of interest during the
compile. I think there are actually some serious problems introduced
because of its runtime nature, but nevertheless that's how Python is.
Ian
Steven Taschuk
> Steven Taschuk:
> > Hm. I'm not sure I understand what would make such a language not
> > Python.
>
> I believe apologies are due. It appears I misread flippancy into
> your response, so this one of mine is now more serious.
On the contrary! No apology necessary at all, and you correctly
detected flippancy. (Also see sig.) I'm just valiantly trying to
bat the nit back.
[my claim that generators were a bigger change than silly __and__]
> I don't think generators are that large a change. Pre-generators I
> wrote a lot of code which looked like this
[iterator example]
> Conceptionally then, I see yield (on top of the iterator protocol)
> as a great way to simplify building an existing idiom. [...]
> But Python the language did let me get that feel, even in the
> 1.4 days, and adding generators to Python has not affected
> that feel.
[comparison of silly __and__'s arguments and code blocks]
> Were Python to have code blocks, then it should be applied
> more generally to Python code. Consider the if/else PEP of
> a couple months ago. With code blocks, the following would
> be possible
>
> ifelse(a != 0, 1/a, None)
>
> which would pass the 1/a and None as independent code
> blocks, to be executed based on the evaluation of a != 0.
Ah, you argue well.
I agree that the canonical use of generators -- to write iterators
-- is merely an improvement of a previously existing idiom, and
does not in itself change the feeling of the language.
But generators' suspend/resume powers let them be used far more
generally. PEP 288 alludes to some such uses; see also David
Mertz's developerworks articles on using generators to implement
coroutines.
(I have occasionally thought of using Mertz's technique for
*every* kind of call, not merely coroutine calls. If (say, by
bytecode translation) we turned every call
foo(*args, **kwargs)
into
yield foo, args, kwargs
every function would become a generator. This would allow the
call semantics to be implemented in Python, in what Mertz calls a
scheduler. You could, for example, build microthreads this way;
if generators could be copied, you could even build continuations!
At least I think so; this is all just a vague idea I have, not
supported by any detailed knowledge of Python internals. There
are many details to be worked out, e.g., how return values are
provided to the "calling" function, if the call "returns". (PEP
288 would provide one mechanism.) There are also efficiency
questions, since e.g. every use of + is potentially a call.)
In other words, the suspend/resume powers of generators open up
all sorts of interesting possibilities which would be difficult to
implement in pre-generator Pythons, and imho these capabilities
can make for significant changes to the feel of the language.
I feel that generators, conceived this broadly, are just as
significant a change as code blocks would be. (I'm backing down
from my previous claim that they are *more* of a change than the
code blocks implied by the arguments to my silly __and__; now I
claim only that they're of about the same magnitude.)
--
Steven Taschuk stas...@telusplanet.net
"I'm always serious, never more so than when I'm being flippant."
-- _Look to Windward_, Iain M. Banks