Python's super() considered super!
Raymond Hettinger
super().
One of the reviewers, David Beazley, said, "Wow, that's really
great! I see this becoming the definitive post on the subject"
The direct link is:
http://rhettinger.wordpress.com/2011/05/26/super-considered-super/
It would also be great if some of you would upvote it on HackerNews.
Raymond Hettinger
-----------------------
follow my python tips on twitter: @raymondh
Raymond Hettinger
Here's a link to the super() how-to-guide and commentary: bit.ly/
iFm8g3
Raymod
Dotan Cohen
Is that the same link as in the OP? I don't click on blind links, so
if it isn't then please post a direct link. Thanks.
--
Dotan Cohen
Ian Kelly
> On Thu, May 26, 2011 at 19:39, Raymond Hettinger <pyt...@rcn.com> wrote:
> if it isn't then please post a direct link. Thanks.
It's a link to ycombinator:
Dotan Cohen
> It's a link to ycombinator:
>
> http://news.ycombinator.com/item?id=2588262
>
Thanks.
Terry Reedy
> On Thu, May 26, 2011 at 19:39, Raymond Hettinger<pyt...@rcn.com> wrote:
> if it isn't then please post a direct link. Thanks.
It is a link to HackerNews
http://news.ycombinator.com/item?id=2588262
--
Terry Jan Reedy
Ben Finney
> I just posted a tutorial and how-to guide for making effective use of
> super().
Thanks very much! We need articles like this.
Raymond Hettinger <pyt...@rcn.com> writes:
> Here's a link to the super() how-to-guide and commentary: bit.ly/
> iFm8g3
We also, though, need *real* URLs. Blind URLs through obfuscation
services have their uses, but surely not in a forum like this. The real
URL is <URL:http://news.ycombinator.com/item?id=2588262>.
--
\ “… it's best to confuse only one issue at a time.” —Brian W. |
`\ Kernighan and Dennis M. Ritchie, _The C programming language_, |
_o__) 1988 |
Ben Finney
Raymond Hettinger
> We also, though, need *real* URLs. Blind URLs through obfuscation
> services have their uses, but surely not in a forum like this. The real
> URL is <URL:http://news.ycombinator.com/item?id=2588262>.
Fair enough. I had copied the link from Jesse's tweet (where shorter
is better).
Hope you enjoyed the post.
Raymond
Ben Finney
> Hope you enjoyed the post.
I certainly did.
But I'm not better enlightened on why ‘super’ is a good thing. The
exquisite care that you describe programmers needing to maintain is IMO
just as much a deterrent as the super-is-harmful essay.
--
\ “If you continue running Windows, your system may become |
`\ unstable.” —Microsoft, Windows 95 bluescreen error message |
_o__) |
Ben Finney
Michele Simionato
> The exquisite care that you describe programmers needing to maintain is IMO
> just as much a deterrent as the super-is-harmful essay.
Worth quoting. Also I think this article may encourage naive programmers along the dark path of cooperative multiple inheritance, when they could instead use better designs.
Steven D'Aprano
> Raymond Hettinger <pyt...@rcn.com> writes:
>
>> Hope you enjoyed the post.
>
> I certainly did.
>
> But I'm not better enlightened on why ‘super’ is a good thing.
Perhaps Raymond assumed that by now everybody knows that multiple
inheritance in Python that doesn't use super is buggy. super() was
introduced in version 2.2 in order to overcome bugs in MI, making it
about 8 years old now.
(Technically, it's only MI with diamond-shaped inheritance, but that
applies to *all* new-style classes. If you're writing multiple
inheritance in Python 3 without using super, your code is a land-mine
waiting to go off. If you're writing single inheritance, it's *still* a
land-mine, just waiting for some poor caller to use it in a MI context.)
But I do agree with you in that I expected to see at least some
discussion of why super should be actively preferred over calling the
parent class directly.
> The
> exquisite care that you describe programmers needing to maintain is IMO
> just as much a deterrent as the super-is-harmful essay.
I don't see that Raymond describes anything needing "exquisite care".
It's more common sense really: ensure that your method signature and that
of your parents' match, plus good work-arounds for when they don't.
Anyone using inheritance is almost certainly 98% of the way there, unless
they're writing classes like this and wondering why they don't work :)
class MyBrokenList(list):
def __len__(self):
n = list.__len__(self, extra=42)
return n + 1
def __getitem__(self, i):
return list.__getitem__(self) + 1
def last_item(self):
return list.last_item(self) + 1
I was thrilled to learn a new trick, popping keyword arguments before
calling super, and wondered why I hadn't thought of that myself. How on
earth did I fail to realise that a kwarg dict was mutable and therefore
you can remove keyword args, or inject new ones in?
Given the plethora of articles that take a dim, if not outright negative,
view of super, it is good to see one that takes a positive view. Thanks
Raymond!
--
Steven
Duncan Booth
> I was thrilled to learn a new trick, popping keyword arguments before
> calling super, and wondered why I hadn't thought of that myself. How on
> earth did I fail to realise that a kwarg dict was mutable and therefore
> you can remove keyword args, or inject new ones in?
>
Probably because most of the time it is better to avoid mutating kwargs.
Instead of popping an argument you simply declare it as a named argument in
the method signature. Instead of injecting new ones you can pass them as
named arguments.
def foo(x=None, **kwargs):
bar(y=2, **kwargs)
def bar(**kwargs):
print(kwargs)
>>> foo(x=1, z=3)
{'y': 2, 'z': 3}
>>> foo(x=1, y=2, z=3)
Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
foo(x=1, y=2, z=3)
File "<pyshell#4>", line 2, in foo
bar(y=2, **kwargs)
TypeError: bar() got multiple values for keyword argument 'y'
--
Duncan Booth http://kupuguy.blogspot.com
sturlamolden
> I just posted a tutorial and how-to guide for making effective use of
> super().
>
> One of the reviewers, David Beazley, said, "Wow, that's really
> great! I see this becoming the definitive post on the subject"
>
> The direct link is:
>
> http://rhettinger.wordpress.com/2011/05/26/super-considered-super/
I really don't like the Python 2 syntax of super, as it violates
the DRY principle: Why do I need to write super(type(self),self)
when super() will do? Assuming that 'self' will always be named
'self' in my code, I tend to patch __builtins__.super like this:
import sys
def super():
self = sys._getframe().f_back.f_locals['self']
return __builtins__.super(type(self),self)
This way the nice Python 3.x syntax can be used in Python 2.x.
Sturla
Mel
> I really don't like the Python 2 syntax of super, as it violates
> the DRY principle: Why do I need to write super(type(self),self)
> when super() will do? Assuming that 'self' will always be named
> 'self' in my code, I tend to patch __builtins__.super like this:
>
> import sys
> def super():
> self = sys._getframe().f_back.f_locals['self']
> return __builtins__.super(type(self),self)
>
> This way the nice Python 3.x syntax can be used in Python 2.x.
Python causes trouble by letting the users get at the internals, but things
like this make it worthwhile.
Mel.
Steven D'Aprano
Only if by "worthwhile" you mean "buggy as hell".
>>> import sys
>>>
>>> def super():
... self = sys._getframe().f_back.f_locals['self']
... return __builtins__.super(type(self),self)
...
>>> class A(object):
... def __init__(self):
... super().__init__()
...
>>> class B(A):
... def __init__(self):
... super().__init__()
...
>>> a = A()
>>> b = B()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __init__
[...]
File "<stdin>", line 3, in __init__
RuntimeError: maximum recursion depth exceeded
Do not use super(type(self), self), because it does not do what you think
it does:
b = B() calls B.__init__(self)
... which calls super(type(self), self) = super(B, self)
... which calls A.__init__(self)
... which calls super(type(self), self) = super(B, self) *not* A
... which loops forever
type(self) does not return B inside B methods and A inside A methods, it
returns the class of the instance.
--
Steven
Michele Simionato
super(type(self),self) in Python 2 is NOT equivalent to super()
in Python 3 is telling something.
Duncan Booth
Oh dear, you haven't thought this one through.
After your code above, try this:
>>> class A(object):
def foo(self):
print "A.foo"
>>> class B(A):
def foo(self):
print "B.foo"
super().foo()
>>> B().foo()
B.foo
A.foo
So far so good.
Now try this:
>>> class C(B): pass
>>> C().foo()
... infinite recursion follows ...
Oops. There's a reason why Python 2 requires you to be explicit about
the class; you simply cannot work it out automatically at run time.
Python 3 fixes this by working it out at compile time, but for Python 2
there is no way around it.
harrismh777
>> Python causes trouble by letting the users get at the internals, but
>> > things like this make it worthwhile.
> Only if by "worthwhile" you mean "buggy as hell".
I *don't* believe this... the king of metaphors and bogus analogies
has come up with 'buggy as hell' !!?
No no, you might have buggy as grubs-under-a-damp-log, or buggy as
'moths-around-an-incandecent-lamp' , or you could have 'hot-as-hell,'
but 'buggy-as-hell' just doesn't say what needs say'in...
... I'm just saying....
:)
sturlamolden
> >>> class C(B): pass
> >>> C().foo()
>
> ... infinite recursion follows ...
That's true :(
sturlamolden
> Oops. There's a reason why Python 2 requires you to be explicit about
> the class; you simply cannot work it out automatically at run time.
> Python 3 fixes this by working it out at compile time, but for Python 2
> there is no way around it.
Then it should be a keyword, not a function.
Sturla
Steven D'Aprano
Why? The fault is not that super is a function, or that you monkey-
patched it, or that you used a private function to do that monkey-
patching. The fault was that you made a common, but silly, mistake when
reasoning about type(self) inside a class.
I made the same mistake: assume that type(self) will always be the same
class as that the method is defined in. But of course it won't be. With
the luxury of hindsight, it is a silly mistake to make, but I promise you
that you're not the first, and won't be the last, to make it.
--
Steven
Ethan Furman
And the above error is exactly why you don't want to use named arguments
in MI -- because you don't know in what order the methods will be
called, you cannot know which named arguments to supply to the method
that super() will call next.
~Ethan~
Prasad, Ramit
>>services have their uses, but surely not in a forum like this. The real
>>URL is <URL:http://news.ycombinator.com/item?id=2588262>.
Ramit
Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423
This communication is for informational purposes only. It is not
intended as an offer or solicitation for the purchase or sale of
any financial instrument or as an official confirmation of any
transaction. All market prices, data and other information are not
warranted as to completeness or accuracy and are subject to change
without notice. Any comments or statements made herein do not
necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
and affiliates.
This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law. If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. Although this transmission and any
attachments are believed to be free of any virus or other defect
that might affect any computer system into which it is received and
opened, it is the responsibility of the recipient to ensure that it
is virus free and no responsibility is accepted by JPMorgan Chase &
Co., its subsidiaries and affiliates, as applicable, for any loss
or damage arising in any way from its use. If you received this
transmission in error, please immediately contact the sender and
destroy the material in its entirety, whether in electronic or hard
copy format. Thank you.
Please refer to http://www.jpmorgan.com/pages/disclosures for
disclosures relating to European legal entities.
Ian Kelly
> I just posted a tutorial and how-to guide for making effective use of
> super().
I posted this already on the HackerNews thread but it seems to have
largely gone unnoticed, so I'm reposting it here.
It seems to me that the example of combining built-in dictionary
classes is naively optimistic. For starters, OrderedDict, as it
happens, does not use super! It calls the dict super-class methods
directly. Since dict luckily happens to be the next class in the MRO,
this doesn't really matter for the purpose of this example, but I can
envision a scenario where some plucky programmer inherits from both
OrderedCounter and some other dict subclass, and the result doesn't
work because OrderedDict does the wrong thing.
And OrderedDict isn't the only one. Maybe for some reason I would like
to have an OrderedCounter where all the counts default to 42. So I do
this:
class DefaultOrderedCounter(defaultdict, OrderedCounter):
pass
doc = DefaultOrderedCounter(lambda: 42)
doc.update('abracadabra')
Which results in:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\python32\lib\collections.py", line 507, in update
_count_elements(self, iterable)
File "c:\python32\lib\collections.py", line 63, in __setitem__
self.__map[key] = link = Link()
AttributeError: 'DefaultOrderedCounter' object has no attribute
'_OrderedDict__map'
Whoops! Apparently defaultdict doesn't use super either. Of course a
better way to do this would be to subclass OrderedCounter and just
override the __missing__ method by hand, but that's not the point.
The article goes into "How to Incorporate a Non-cooperative Class",
which basically says "wrap it up in a proxy class". But that's not
really going to work here, since the result would be two separate
dicts, with the defaultdictwrapper methods operating on one dict, and
the other methods operating on the other.
Chris Angelico
<ramit....@jpmchase.com> wrote:
>>>We also, though, need *real* URLs. Blind URLs through obfuscation
>>>services have their uses, but surely not in a forum like this. The real
>>>URL is <URL:http://news.ycombinator.com/item?id=2588262>.
>
I've seen FAR more dead links than dead URL shortening services. It's
a lot more likely that the destination will go down than that the
tinyurl service will lose its data.
If you're worried about where you're going, grab a URL renderer;
TinyURL.com has "preview mode" which you can set with a cookie, and
for others, all you need is something which takes a URL off the
clipboard, requests it, gets the Location: header, and puts that on
the clipboard for you. I coded such a facility into my MUD client
(RosMud), because shortened URLs are important when lines are limited
to 80 characters (less some overhead); it'd be quite easy to build a
little Python-GTK or Python-TK app that gives you a nice window and
makes it easy.
Chris Angelico
Chris Angelico
> It seems to me that the example of combining built-in dictionary
> classes is naively optimistic.
So... Can anyone offer a non-trivial example of multiple inheritance
that *doesn't* have pitfalls? From what I've seen, MI always seems to
require cooperation from the authors of all involved classes. It may
be a useful tool when you control everything, but whenever you use
someone else's code, there seems to be this massive barrier of risk
(if that makes sense). For the DefaultOrderedCounter, I would be
strongly inclined to inherit singly, and then manually implement the
other half (whichever half is easier); in this case that happens to be
trivial (override __missing__), but even were it not, it would be a
means of being certain that things won't break.
Chris Angelico
John Nagle
> On Sat, May 28, 2011 at 4:31 AM, Ian Kelly<ian.g...@gmail.com> wrote:
>> It seems to me that the example of combining built-in dictionary
>> classes is naively optimistic.
>
> So... Can anyone offer a non-trivial example of multiple inheritance
> that *doesn't* have pitfalls? From what I've seen, MI always seems > to require cooperation from
the authors of all involved classes.
Good point.
Multiple inheritance is messy enough when the structure is just
a tree. When the structure is allowed to be a directed acyclic
graph, the whole thing just gets too complicated.
It doesn't even necessarily do what you want. If, say, you
have two classes that need dictionaries, and were implemented
by inheriting from "dict", a class that imports both has one
"dict", not two - something that was not anticipated in the
design of the original classes. That ought to be an error,
not a single "dict" shared by two unconnected classes.
What causes this kind of mess is a determination not to
report anything as an error if it can be given some kind of
meaningful semantics, even if the semantics have marginal
value. That's the kind of thinking which leads to
[1,2] * 2
returning
[1,2,1,2]
John Nagle
Stefan Behnel
> On Fri, 27 May 2011 08:31:40 -0700, sturlamolden wrote:
>
>> On 27 Mai, 17:05, Duncan Booth<duncan.bo...@invalid.invalid> wrote:
>>
>>> Oops. There's a reason why Python 2 requires you to be explicit about
>>> the class; you simply cannot work it out automatically at run time.
>>> Python 3 fixes this by working it out at compile time, but for Python 2
>>> there is no way around it.
>>
>> Then it should be a keyword, not a function.
>
> Why?
I think Sturla is referring to the "compile time" bit. CPython cannot know
that the builtin super() will be called at runtime, even if it sees a
"super()" function call.
CPython doesn't evaluate the super call at compile time, it only keeps a
reference to the surrounding class in the code object of the method. So
super() without arguments basically inherits the class argument from the
context the method was found in at compile time. This has two quirks:
1) Copying a method from one class to another keeps the original context.
So the class argument to super() is basically fixed at compile time,
regardless of the class the method will be executed on at runtime.
2) The class is only kept as a reference when CPython sees a function call
that looks like "super" at compile time, which isn't much more than a
heuristic.
The PEP doesn't mention the first issue, but it is actually explicit about
the second issue:
http://www.python.org/dev/peps/pep-3135/
"""
While super is not a reserved word, the parser recognizes the use of super
in a method definition and only passes in the __class__ cell when this is
found. Thus, calling a global alias of super without arguments will not
necessarily work.
"""
And the prove:
>>> _super = super
>>> class T(object):
... def test(self): print('TEST')
...
>>> class B(T):
... def meth(self): _super().test()
...
>>> B().meth()
Traceback (most recent call last):
SystemError: super(): __class__ cell not found
I assume this is done in order to reduce the chance of accidentally keeping
a class object alive for eternity, only because a method was originally
defined therein that inherits the class reference in its code object. So
it's a tradeoff between memory overhead and usability issues.
While I think that the tradeoff is generally ok, I agree with Sturla that a
keyword would have been the correct solution, whereas this is a clear
"works only in the common cases" approach.
Stefan
Ethan Furman
folks don't appreciate that. Ask anyone about meta-classes and their
eyes bug-out, but MI? Simple! NOT.
On the other hand, perhaps the docs should declare that the built-in
objects are not designed for MI, so that that, at least, would be one
less bug waiting to happen.
~Ethan~
sturlamolden
> I think Sturla is referring to the "compile time" bit. CPython cannot know
> that the builtin super() will be called at runtime, even if it sees a
> "super()" function call.
Yes. And opposite: CPython cannot know that builtin super() is not
called,
even if it does not see the name 'super'. I can easily make foo()
alias super().
In both cases, the cure is a keyword -- or make sure that __class__
is always defined.
If super is to be I keyword, we could argue that self and cls should
be
keywords as well, and methods should always be bound. That speaks in
favour
of a super() function. But then it should always be evaluated at run-
time,
without any magic from the parser.
Magic functions belong in Perl, not Python.
Sturla
Ryan Kelly
> sturlamolden <sturla...@yahoo.no> wrote:
> > I really don't like the Python 2 syntax of super, as it violates
> > the DRY principle: Why do I need to write super(type(self),self)
> > when super() will do? Assuming that 'self' will always be named
> > 'self' in my code, I tend to patch __builtins__.super like this:
> >
> > import sys
> > def super():
> > self = sys._getframe().f_back.f_locals['self']
> > return __builtins__.super(type(self),self)
> >
> > This way the nice Python 3.x syntax can be used in Python 2.x.
> >
> >
> Oh dear, you haven't thought this one through.
>
>
> >>> C().foo()
> ... infinite recursion follows ...
>
> Oops. There's a reason why Python 2 requires you to be explicit about
> the class; you simply cannot work it out automatically at run time.
> Python 3 fixes this by working it out at compile time, but for Python 2
> there is no way around it.
Oh? There's not much that can't be done at runtime if you're willing to
work hard enough. Let me propose the following awful awful hack:
import sys
_builtin_super = __builtins__.super
_sentinel = object()
def _auto_super(typ=_sentinel,type_or_obj=_sentinel):
"""Automagically call correct super() at runtime"""
# Infer the correct call if used without arguments.
if typ is _sentinel:
# We'll need to do some frame hacking.
f = sys._getframe(1)
# Get the first positional argument of the function.
type_or_obj = f.f_locals[f.f_code.co_varnames[0]]
# Get the MRO for investigation
try:
mro = type_or_obj.__mro__
except AttributeError:
try:
mro = type_or_obj.__class__.__mro__
except AttributeError:
raise RuntimeError("super() used with old-style class")
# Now, find the class owning the currently-executing method.
for typ in mro:
for meth in typ.__dict__.itervalues():
if not isinstance(meth,type(_auto_super)):
continue
if meth.func_code is f.f_code:
# Aha! Found you.
break
else:
continue
break
else:
raise RuntimeError("super() called outside a method")
# Now just dispatch to builtin super.
if type_or_obj is not _sentinel:
return _builtin_super(typ,type_or_obj)
return _builtin_super(typ)
Now, try is with the following:
class Base(object):
def hello(self,msg):
print "hello", msg
class Sub1(Base):
def hello(self,msg):
print "gunna say it"
super().hello(msg)
class Sub2(Base):
def hello(self,msg):
print "yes I am"
super().hello(msg)
class Diamond(Sub1,Sub2):
def hello(self,msg):
print "here we go..."
super().hello(msg)
d = Diamond()
d.hello("autosuper!")
And you get the expected output:
here we go...
gunna say it
yes I am
hello autosuper!
There may well be some cases where this breaks down, but it seems to do
the right thing in simple cases.
Not that I'm recommending anyone use this, of course...
Ryan
--
Ryan Kelly
http://www.rfk.id.au | This message is digitally signed. Please visit
ry...@rfk.id.au | http://www.rfk.id.au/ramblings/gpg/ for details
sturlamolden
+comp.lang.pyt...@pearwood.info> wrote:
> Why? The fault is not that super is a function, or that you monkey-
> patched it, or that you used a private function to do that monkey-
> patching. The fault was that you made a common, but silly, mistake when
> reasoning about type(self) inside a class.
That was indeed a silly mistake, but not what I am talking about.
See Stefan's reponse.
Sturla
Thomas Rachel
> Yes. And opposite: CPython cannot know that builtin super() is not
> called,
> even if it does not see the name 'super'. I can easily make foo()
> alias super().
Another alternative would have been to make use of __xxx magic.
If every class had an "automatically available" attribute, e. g.
__<classname>_classname which thus could be accessed via __classname
from inside, keeping the 2.x syntax would have been the best, using
super(__classname, self).
> In both cases, the cure is a keyword -- or make sure that __class__
> is always defined.
>
> If super is to be I keyword, we could argue that self and cls should
> be keywords as well, and methods should always be bound. That speaks in
> favour of a super() function. But then it should always be evaluated at run-
> time, without any magic from the parser.
>
> Magic functions belong in Perl, not Python.
ACK.
Thomas
Duncan Booth
>>>>> foo(x=1, y=2, z=3)
>> Traceback (most recent call last):
>> File "<pyshell#8>", line 1, in <module>
>> foo(x=1, y=2, z=3)
>> File "<pyshell#4>", line 2, in foo
>> bar(y=2, **kwargs)
>> TypeError: bar() got multiple values for keyword argument 'y'
>
> And the above error is exactly why you don't want to use named arguments
> in MI -- because you don't know in what order the methods will be
> called, you cannot know which named arguments to supply to the method
> that super() will call next.
>
No, it just means you have to be careful how you do it: if you want to pass
'y' to a super call then you must pop it out of the arguments. This can be
done as easily with a named argument as with an explicit 'pop()' on kwargs.
What using a named argument does it to change the default action from
silently overwriting any argumement passing up the chain to raising an
error if you attempt to overwrite it without first checking whether it
exists.