Add __parent__ to all classes, functions, and modules

[Neil Girdhar]

Many classes, functions, and modules are defined within the context of another class, function, or module thereby forming a mathematical forest of declarations.  It is possible to walk the descendants using __dict__ (for classes and modules), but not the ancestors.  I propose adding __parent__ that would be filled at the same time that __qualname__ is filled in.  One concrete use case for __parent__ is allowing a method decorator to call super, which is currently impossible because the class in which the method has been defined is not available to the decorator.  This way, one could write:

def ensure_finished(iterator):
    try:
        next(iterator)
    except StopIteration:
        return
    else:
        raise RuntimeError


def derived_generator(method):
    def new_method(self, *args, **kwargs):
        x = method(self, *args, **kwargs)
        y = getattr(super(method.__parent__, self), method.__name__)\
            (*args, **kwargs)

        for a, b in zip(x, y):
            assert a is None and b is None
            yield

        ensure_finished(x)
        ensure_finished(y)

This is currently impossible to implement without restating the class name every time the decorator is used as far as I know.

Best,

Neil

[Benjamin Peterson]

Neil Girdhar <mistersheik@...> writes:

>
> Many classes, functions, and modules are defined within the context of
another class, function, or module thereby forming a mathematical forest of
declarations.  It is possible to walk the descendants using __dict__ (for
classes and modules), but not the ancestors.  I propose adding __parent__
that would be filled at the same time that __qualname__ is filled in.

This is unlikely to work.

1) It turns basically everything into a cycle.
2) __qualname__ is determined strictly from syntax, whereas __parent__ could
not be. For example, what happens if I take a method from one class and set
it on another? __parent__ would not be well-defined.

Regards,
Benjamin

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Neil Girdhar]

On Sun, Oct 5, 2014 at 2:09 PM, Benjamin Peterson <benj...@python.org> wrote:

Neil Girdhar <mistersheik@...> writes:

>
> Many classes, functions, and modules are defined within the context of
another class, function, or module thereby forming a mathematical forest of
declarations.  It is possible to walk the descendants using __dict__ (for
classes and modules), but not the ancestors.  I propose adding __parent__
that would be filled at the same time that __qualname__ is filled in.

This is unlikely to work.

1) It turns basically everything into a cycle.

Why a cycle? 

2) __qualname__ is determined strictly from syntax, whereas __parent__ could
not be. For example, what happens if I take a method from one class and set
it on another? __parent__ would not be well-defined.

I'm suggesting that parent be determined purely from declaration.  If you copy something, neither qualname nor parent would change unless you change them. 

Regards,
Benjamin


_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--

---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/94fTkAkjhCo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Benjamin Peterson]

Neil Girdhar <mistersheik@...> writes:

>
>
>
> On Sun, Oct 5, 2014 at 2:09 PM, Benjamin Peterson <benjamin <at>
python.org> wrote:

> Neil Girdhar <mistersheik <at> ...> writes:
> >
> > Many classes, functions, and modules are defined within the context of
> another class, function, or module thereby forming a mathematical forest of
> declarations.  It is possible to walk the descendants using __dict__ (for
> classes and modules), but not the ancestors.  I propose adding __parent__
> that would be filled at the same time that __qualname__ is filled in.This
is unlikely to work.
> 1) It turns basically everything into a cycle.
>
>
> Why a cycle? 

Because, for example, the class would reference methods, which would
reference the class.

>
>
> 2) __qualname__ is determined strictly from syntax, whereas __parent__ could
> not be. For example, what happens if I take a method from one class and set
> it on another? __parent__ would not be well-defined.
>
>
> I'm suggesting that parent be determined purely from declaration.  If you
copy something, neither qualname nor parent would change unless you change
them. 

It would mean your trick with super would only work for some methods.

[Neil Girdhar]

On Sun, Oct 5, 2014 at 5:11 PM, Benjamin Peterson <benj...@python.org> wrote:

Neil Girdhar <mistersheik@...> writes:

>
>
>
> On Sun, Oct 5, 2014 at 2:09 PM, Benjamin Peterson <benjamin <at>
python.org> wrote:
> Neil Girdhar <mistersheik <at> ...> writes:
> >
> > Many classes, functions, and modules are defined within the context of
> another class, function, or module thereby forming a mathematical forest of
> declarations.  It is possible to walk the descendants using __dict__ (for
> classes and modules), but not the ancestors.  I propose adding __parent__
> that would be filled at the same time that __qualname__ is filled in.This
is unlikely to work.
> 1) It turns basically everything into a cycle.
>
>
> Why a cycle? 

Because, for example, the class would reference methods, which would
reference the class.

Right.  I don't see what's wrong with that.  This already happens with the __module__ member and the module's immediate children.

>
>
> 2) __qualname__ is determined strictly from syntax, whereas __parent__ could
> not be. For example, what happens if I take a method from one class and set
> it on another? __parent__ would not be well-defined.
>
>
> I'm suggesting that parent be determined purely from declaration.  If you
copy something, neither qualname nor parent would change unless you change
them. 

It would mean your trick with super would only work for some methods.

It would only work for that are decorated in the usual way using @blah on methods defined in the class.  If someone wants to copy that method somewhere else then they'll have to update __parent__ themselves.

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Guido van Rossum]

This is a clever idea but I doubt it's worth adding.

The pattern proposed in the example in the first post doesn't strike me as very readable -- though you don't give any examples of *using* the decorator from the example, so it's a little difficult to imagine what the use case would be.

Adding __parent__ would have to happen as an extra pass once the class is defined -- the decorator (at least its outer part) runs at the time that the method is being defined as a plain function object, and at that time the class object hasn't been created yet. This also means that some decorators can't use the __parent__ attribute (because the outer function of the decorator runs before the object to be stored in __parent__ has been created).

If you really want to explore this idea further, you can probably define a metaclass that adds a __parent__ attribute to the function objects in the class __dict__ when the class is being created. Then you can experiment with using the pattern in some real code. Please report back here with some examples where the presence of __parent__ lets you write cleaner code.

An alternative that wouldn't even require a metaclass would be to write a helper function that looks the class object up by name after parsing __qualname__. There are some limitations to this, e.g. classes defined inside functions -- but that's already a suspect pattern anyway.

--Guido van Rossum (python.org/~guido)

[Steven D'Aprano]

On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote:

> An alternative that wouldn't even require a metaclass would be to write a
> helper function that looks the class object up by name after parsing
> __qualname__. There are some limitations to this, e.g. classes defined
> inside functions -- but that's already a suspect pattern anyway.

I understand that "suspect pattern" doesn't mean "don't ever do this",
but I wonder what aspect of defining classes inside a function is
suspect. I think the suspect part is that each invocation of the
function creates a new class which (at least in the naive case) has the
same name, same functionality, and looks identical to the casual
glance but actually isn't. So given:

def maker():
class MyClass: pass
return MyClass

x = maker()()
y = maker()()

assert isinstance(x, type(y))
assert isinstance(y, type(x))


both assertions will fail, which may confuse some people. There's
probably also implications for pickling as well. Are these the only
"suspect" parts of defining classes inside functions, or have I missed
something else?



--
Steven

[Neil Girdhar]

On Sunday, October 5, 2014 5:43:33 PM UTC-4, Guido van Rossum wrote:

This is a clever idea but I doubt it's worth adding.

The pattern proposed in the example in the first post doesn't strike me as very readable -- though you don't give any examples of *using* the decorator from the example, so it's a little difficult to imagine what the use case would be.

The idea is that I have a caller who is making changes to a network of nodes in steps.  The caller tells the link to do phase 1, then makes some changes, then tells the link to do phase 2, then makes some changes, etc.  Each time, the link can memoize some values from the network, send signals, etc.  Therefore, I did not want to have the phases be different methods since the memos would then have to be stored on the object.  Instead the phases are implemented using the generator pattern (that is, each phase is separated by yield) and the caller simply iterates over the generator as it makes its changes. The problem is that in addition to what is implemented by the method, the method wants to also delegate to super.  So we need to iterate over the generator returned by the superclass.  This pattern is easy to follow, but it would be nice to wrap that up in a decorator so that the method-writer can forget about this detail. 

 

Adding __parent__ would have to happen as an extra pass once the class is defined -- the decorator (at least its outer part) runs at the time that the method is being defined as a plain function object, and at that time the class object hasn't been created yet. This also means that some decorators can't use the __parent__ attribute (because the outer function of the decorator runs before the object to be stored in __parent__ has been created).

Couldn't __parent__ be filled in when the class is complete by the default metaclass (is that just "type"?)  The generator doesn't need to actually do anything with the parent member.  The lookup is happening in the created method, so I think that's at call time, at which point the attribute will be filled in.  I could have done the same thing with qualname, right?

 

If you really want to explore this idea further, you can probably define a metaclass that adds a __parent__ attribute to the function objects in the class __dict__ when the class is being created. Then you can experiment with using the pattern in some real code. Please report back here with some examples where the presence of __parent__ lets you write cleaner code.

The problem is that such a metaclass would add the member to the decorated method, and unfortunately, methods and functions don't have access to themselves and their attribtes at runtime as far as I know.  So I'd have to look inside the members and add the __parent__ attribute to "method" attributes of the decorated methods, which is a bit weird.   

An alternative that wouldn't even require a metaclass would be to write a helper function that looks the class object up by name after parsing __qualname__. There are some limitations to this, e.g. classes defined inside functions -- but that's already a suspect pattern anyway.

Yes, that works, but it really is very ugly to parse a string and then eval it.

[Guido van Rossum]

On Sun, Oct 5, 2014 at 7:36 PM, Steven D'Aprano <st...@pearwood.info> wrote:

On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote:

> An alternative that wouldn't even require a metaclass would be to write a
> helper function that looks the class object up by name after parsing
> __qualname__. There are some limitations to this, e.g. classes defined
> inside functions -- but that's already a suspect pattern anyway.

I understand that "suspect pattern" doesn't mean "don't ever do this",
but I wonder what aspect of defining classes inside a function is
suspect. I think the suspect part is that each invocation of the
function creates a new class which (at least in the naive case) has the
same name, same functionality, and looks identical to the casual
glance but actually isn't. So given:

def maker():
    class MyClass: pass
    return MyClass

x = maker()()
y = maker()()

assert isinstance(x, type(y))
assert isinstance(y, type(x))


both assertions will fail, which may confuse some people. There's
probably also implications for pickling as well. Are these the only
"suspect" parts of defining classes inside functions, or have I missed
something else?

Those (and their variations, e.g. using these in except clauses) are the main concerns; it also generally bothers me when something "smallish" like a function contains something "larger" like a class, especially when the class is substantial. It's another case of "flat is better than nested".

The only non-evil use of this pattern that I can recall is in unit tests -- even when it is possible (as it usually is) to move the class definition to a spot outside the test method, that doesn't make the test code any clearer. Such classes are usually very small (e.g. overriding just one or two methods) and are only used within the unit test. (That latter property is really the saving grace.)

[Guido van Rossum]

On Sun, Oct 5, 2014 at 7:51 PM, Neil Girdhar <miste...@gmail.com> wrote:

On Sunday, October 5, 2014 5:43:33 PM UTC-4, Guido van Rossum wrote:

This is a clever idea but I doubt it's worth adding.

The pattern proposed in the example in the first post doesn't strike me as very readable -- though you don't give any examples of *using* the decorator from the example, so it's a little difficult to imagine what the use case would be.

The idea is that I have a caller who is making changes to a network of nodes in steps.  The caller tells the link to do phase 1, then makes some changes, then tells the link to do phase 2, then makes some changes, etc.  Each time, the link can memoize some values from the network, send signals, etc.  Therefore, I did not want to have the phases be different methods since the memos would then have to be stored on the object.  Instead the phases are implemented using the generator pattern (that is, each phase is separated by yield) and the caller simply iterates over the generator as it makes its changes. The problem is that in addition to what is implemented by the method, the method wants to also delegate to super.  So we need to iterate over the generator returned by the superclass.  This pattern is easy to follow, but it would be nice to wrap that up in a decorator so that the method-writer can forget about this detail. 

I'm sorry, you'd have to point to the actual code if you want anyone to make sense of that description. You are combining so many abstractions in one paragraph that it feels like there must be lots of entirely different ways to map the ideas to code, and you may well just have picked the wrong way.

 

Adding __parent__ would have to happen as an extra pass once the class is defined -- the decorator (at least its outer part) runs at the time that the method is being defined as a plain function object, and at that time the class object hasn't been created yet. This also means that some decorators can't use the __parent__ attribute (because the outer function of the decorator runs before the object to be stored in __parent__ has been created).

Couldn't __parent__ be filled in when the class is complete by the default metaclass (is that just "type"?)

I think that's what I was saying. :-)
 

 The generator doesn't need to actually do anything with the parent member.  The lookup is happening in the created method, so I think that's at call time, at which point the attribute will be filled in.  I could have done the same thing with qualname, right?

Eh? Anyway, __qualname__ does not have this problem -- the class *name* is known by the time we get to 'def'.

 

If you really want to explore this idea further, you can probably define a metaclass that adds a __parent__ attribute to the function objects in the class __dict__ when the class is being created. Then you can experiment with using the pattern in some real code. Please report back here with some examples where the presence of __parent__ lets you write cleaner code.

The problem is that such a metaclass would add the member to the decorated method, and unfortunately, methods and functions don't have access to themselves and their attribtes at runtime as far as I know.  So I'd have to look inside the members and add the __parent__ attribute to "method" attributes of the decorated methods, which is a bit weird.

That's a fair point. But the default metaclass (i.e. type) would encounter the same problem. So how *do* you imagine this should work? The fact remains that the class isn't constructed until all the method definitions have been executed, and the namespace in which the methods are defined is what's passed to type() to construct the class object (with the class name and the list of base classes).

 

An alternative that wouldn't even require a metaclass would be to write a helper function that looks the class object up by name after parsing __qualname__. There are some limitations to this, e.g. classes defined inside functions -- but that's already a suspect pattern anyway.

Yes, that works, but it really is very ugly to parse a string and then eval it.

Which you should take as a hint that your problem is looking for a different solution. :-)

[Neil Girdhar]

I'm not sure, but I thought it was only for the standard library to define such names since PEP8 is clear that:

 __double_leading_and_trailing_underscore__: "magic" objects or attributes that live in user-controlled namespaces. E.g. __init__, __import__ or __file__. Never invent such names; only use them as documented.

On Mon, Oct 6, 2014 at 2:35 AM, Wichert Akkerman <wic...@wiggy.net> wrote:

On 05 Oct 2014, at 19:27, Neil Girdhar <miste...@gmail.com> wrote:

> Many classes, functions, and modules are defined within the context of another class, function, or module thereby forming a mathematical forest of declarations.  It is possible to walk the descendants using __dict__ (for classes and modules), but not the ancestors.  I propose adding __parent__ that would be filled at the same time that __qualname__ is filled in.

Independent from if this is a good idea or not: both Zope and Pyramid already make heavy use of __parent__ (and __name__) attributes. This would break those quite badly.

Wichert.

[Andrew Barnert]

On Oct 5, 2014, at 20:04, Guido van Rossum <gu...@python.org> wrote:

On Sun, Oct 5, 2014 at 7:36 PM, Steven D'Aprano <st...@pearwood.info> wrote:

On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote:

> An alternative that wouldn't even require a metaclass would be to write a
> helper function that looks the class object up by name after parsing
> __qualname__. There are some limitations to this, e.g. classes defined
> inside functions -- but that's already a suspect pattern anyway.

I understand that "suspect pattern" doesn't mean "don't ever do this",
but I wonder what aspect of defining classes inside a function is
suspect. I think the suspect part is that each invocation of the
function creates a new class which (at least in the naive case) has the
same name, same functionality, and looks identical to the casual
glance but actually isn't.

…

There's
probably also implications for pickling as well. Are these the only
"suspect" parts of defining classes inside functions, or have I missed
something else?

Those (and their variations, e.g. using these in except clauses) are the main concerns; it also generally bothers me when something "smallish" like a function contains something "larger" like a class, especially when the class is substantial. It's another case of "flat is better than nested".

The only non-evil use of this pattern that I can recall is in unit tests

I agree that returning a class from a function is an uncommon special case, and any code that does so ought to have a good and obvious reason. But to say that such functions are evil doesn't seem right.

What about the functional constructors for namedtuple, or the two kinds of enums? Or a custom kind of enum (strings, bitsets, ...), which the PEP specifically suggests building on top of the stdlib support? Or something more inherently dynamic like a bridge library? Even in cases like generating an ORM from a SQL dump or a client from an RPC discovery call, which could be moved to a separate "pre-compilation" phase of building your application, is a function that generates source code for a class less suspect than one that just generates a class?

[Ethan Furman]

On 10/06/2014 08:07 AM, Andrew Barnert wrote:

> On Oct 5, 2014, at 20:04, Guido van Rossum wrote:
>>
>> The only non-evil use of this pattern that I can recall is in unit tests
>
> I agree that returning a class from a function is an uncommon special case, and any code that does so ought to have a
> good and obvious reason. But to say that such functions are evil doesn't seem right.
>

> What about [the] the two kinds of enums? Or a custom kind of enum (strings,

> bitsets, ...), which the PEP specifically suggests building on top of the stdlib support?

What do you mean? Enums are classes, and the standard way to make new Enum types is by subclassing.

--
~Ethan~

[Guido van Rossum]

On Mon, Oct 6, 2014 at 8:07 AM, Andrew Barnert <abar...@yahoo.com.dmarc.invalid> wrote:

On Oct 5, 2014, at 20:04, Guido van Rossum <gu...@python.org> wrote:

On Sun, Oct 5, 2014 at 7:36 PM, Steven D'Aprano <st...@pearwood.info> wrote:

On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote:

> An alternative that wouldn't even require a metaclass would be to write a
> helper function that looks the class object up by name after parsing
> __qualname__. There are some limitations to this, e.g. classes defined
> inside functions -- but that's already a suspect pattern anyway.

I understand that "suspect pattern" doesn't mean "don't ever do this",
but I wonder what aspect of defining classes inside a function is
suspect. I think the suspect part is that each invocation of the
function creates a new class which (at least in the naive case) has the
same name, same functionality, and looks identical to the casual
glance but actually isn't.

…

There's
probably also implications for pickling as well. Are these the only
"suspect" parts of defining classes inside functions, or have I missed
something else?

Those (and their variations, e.g. using these in except clauses) are the main concerns; it also generally bothers me when something "smallish" like a function contains something "larger" like a class, especially when the class is substantial. It's another case of "flat is better than nested".

The only non-evil use of this pattern that I can recall is in unit tests

I agree that returning a class from a function is an uncommon special case, and any code that does so ought to have a good and obvious reason. But to say that such functions are evil doesn't seem right.

I didn't say *all* uses (or all uses outside unit tests) are evil. I still maintain that most non-test code doing this has a bad smell and is difficult to maintain.

What about the functional constructors for namedtuple, or the two kinds of enums? Or a custom kind of enum (strings, bitsets, ...), which the PEP specifically suggests building on top of the stdlib support? Or something more inherently dynamic like a bridge library? Even in cases like generating an ORM from a SQL dump or a client from an RPC discovery call, which could be moved to a separate "pre-compilation" phase of building your application, is a function that generates source code for a class less suspect than one that just generates a class?

How namedtuple is implemented should be nobody's business, except Raymond's, and it certainly isn't a pattern to be recommended. That's why it's in the stdlib -- so you don't have to write such code yourself. Same for enums. Yes, it *can* be done. But it takes superhuman skills to get it right and it still won't be maintainable (TBH, every time I see the namedtuple implementation I have to resist the urge to rewrite it. :-)

[Andrew Barnert]

Sent from a random iPhone

On Oct 6, 2014, at 10:23, Guido van Rossum <gu...@python.org> wrote:

On Mon, Oct 6, 2014 at 8:07 AM, Andrew Barnert <abar...@yahoo.com.dmarc.invalid> wrote:

On Oct 5, 2014, at 20:04, Guido van Rossum <gu...@python.org> wrote:

On Sun, Oct 5, 2014 at 7:36 PM, Steven D'Aprano <st...@pearwood.info> wrote:

On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote:

> An alternative that wouldn't even require a metaclass would be to write a
> helper function that looks the class object up by name after parsing
> __qualname__. There are some limitations to this, e.g. classes defined
> inside functions -- but that's already a suspect pattern anyway.

I understand that "suspect pattern" doesn't mean "don't ever do this",
but I wonder what aspect of defining classes inside a function is
suspect. I think the suspect part is that each invocation of the
function creates a new class which (at least in the naive case) has the
same name, same functionality, and looks identical to the casual
glance but actually isn't.

…

There's
probably also implications for pickling as well. Are these the only
"suspect" parts of defining classes inside functions, or have I missed
something else?

Those (and their variations, e.g. using these in except clauses) are the main concerns; it also generally bothers me when something "smallish" like a function contains something "larger" like a class, especially when the class is substantial. It's another case of "flat is better than nested".

The only non-evil use of this pattern that I can recall is in unit tests

I agree that returning a class from a function is an uncommon special case, and any code that does so ought to have a good and obvious reason. But to say that such functions are evil doesn't seem right.

I didn't say *all* uses (or all uses outside unit tests) are evil. I still maintain that most non-test code doing this has a bad smell and is difficult to maintain.

OK, I think I just misinterpreted you here. If you're just saying that this is enough of a red flag that you need to pause and prove to yourself (and ideally your code's readers) that it's a "necessary evil" whenever you find yourself writing a function that returns a class, then that makes total sense. Apologies.

What about the functional constructors for namedtuple, or the two kinds of enums? Or a custom kind of enum (strings, bitsets, ...), which the PEP specifically suggests building on top of the stdlib support? Or something more inherently dynamic like a bridge library? Even in cases like generating an ORM from a SQL dump or a client from an RPC discovery call, which could be moved to a separate "pre-compilation" phase of building your application, is a function that generates source code for a class less suspect than one that just generates a class?

How namedtuple is implemented should be nobody's business, except Raymond's, and it certainly isn't a pattern to be recommended. That's why it's in the stdlib -- so you don't have to write such code yourself.

Sure, but I don't think you want every bridge library, dynamic RMDB/GUI/RPC/etc. binding library, etc. to be in the stdlib. _Someone_ has to write them, and the fact that Python makes such code writable (if not easy) is a strength of the language.

Still, I take your point; I wouldn't trust a dynamic binding library that isn't being used by multiple independent projects, because it's so hard to verify by reading that it's gotten all the details right. And a proposal that makes such things slightly easier to write (while still leaving them very hard to get right) probably doesn't add much to the language.

[Guido van Rossum]

No problem! That's what I meant. (Note also that I didn't threaten to remove the feature. :-)
 

What about the functional constructors for namedtuple, or the two kinds of enums? Or a custom kind of enum (strings, bitsets, ...), which the PEP specifically suggests building on top of the stdlib support? Or something more inherently dynamic like a bridge library? Even in cases like generating an ORM from a SQL dump or a client from an RPC discovery call, which could be moved to a separate "pre-compilation" phase of building your application, is a function that generates source code for a class less suspect than one that just generates a class?

How namedtuple is implemented should be nobody's business, except Raymond's, and it certainly isn't a pattern to be recommended. That's why it's in the stdlib -- so you don't have to write such code yourself.

Sure, but I don't think you want every bridge library, dynamic RMDB/GUI/RPC/etc. binding library, etc. to be in the stdlib. _Someone_ has to write them, and the fact that Python makes such code writable (if not easy) is a strength of the language.

Yup. But that code rarely takes the simple form

def make_a_class(parameters):
    ...

    class ClassTemplate:

        def method1(self):
            ...

        def method2(self):
            ...
        ...
    ...

    return C

Still, I take your point; I wouldn't trust a dynamic binding library that isn't being used by multiple independent projects, because it's so hard to verify by reading that it's gotten all the details right. And a proposal that makes such things slightly easier to write (while still leaving them very hard to get right) probably doesn't add much to the language.

Right.

[Neil Girdhar]

You're right — Zope apparently precedes PEP8.

Anyway, I guess I don't need to propose a solution to my problem, but I do think that decorated methods should be able to call super.  Specifically, it's weird that bare super() is able to extract the class name in which it is declared, but there is no other way to do that.  And because there's no other way to do that, a decorator can't call super.

Benjamin's earlier point that copying a method from one class to another would break any attribute storing the "declaration parent class" is already true with any method that calls super.  The call will use the class in which the method was originally defined even if the class you're copying to doesn't derive from there. E.g.,

class A:

    def f(self):

        print("a")

class B(A):

    def f(self):

        super().f()

class C:

    def f(self):

        print("c")

class D:

    pass

D.f = B.f

D().f()  # error, D !< A!!

Right now, a code block has access to globals and locals, but it also has *implicit* access to the defining class via super's default parameters.  Why not get rid of this magic somehow? Maybe Python could expose a pointer to the current function from which an attribute could be set — or some other attribute-based solution. Then super can work by reading this attribute.  This way, if someone copies a method as shown above, she can fix the attribute on the method and super will work as expected.

Best,

Neil

On Mon, Oct 6, 2014 at 5:12 AM, Wichert Akkerman <wic...@wiggy.net> wrote:

On 06 Oct 2014, at 11:09, Neil Girdhar <miste...@gmail.com> wrote:

> I'm not sure, but I thought it was only for the standard library to define such names since PEP8 is clear that:
>
>  __double_leading_and_trailing_underscore__: "magic" objects or attributes that live in user-controlled namespaces. E.g. __init__, __import__ or __file__. Never invent such names; only use them as documented.

Dunder names are indeed received. However Zope has been around for a very very long time and might predate that rule. Pyramid just inherited that naming from Zope.

Wichert.

[Steven D'Aprano]

On Mon, Oct 06, 2014 at 10:23:42AM -0700, Guido van Rossum wrote:

> How namedtuple is implemented should be nobody's business, except
> Raymond's, and it certainly isn't a pattern to be recommended. That's why
> it's in the stdlib -- so you don't have to write such code yourself. Same
> for enums. Yes, it *can* be done. But it takes superhuman skills to get it
> right and it still won't be maintainable (TBH, every time I see the
> namedtuple implementation I have to resist the urge to rewrite it. :-)

Heh :-)

I was curious to see how much code in namedtuple actually needed to be
run through exec, so I forked Raymond's recipe on ActiveState, and
re-wrote with the Inner class as a regular nested class, and only a
two-line __new__ method needing to be run through exec.

http://code.activestate.com/recipes/578918-yet-another-namedtuple/

It passes the minimal tests in the recipe. I haven't taken it any
further.

[Nick Coghlan]

On 7 October 2014 13:54, Steven D'Aprano <st...@pearwood.info> wrote:
> On Mon, Oct 06, 2014 at 10:23:42AM -0700, Guido van Rossum wrote:
>
>> How namedtuple is implemented should be nobody's business, except
>> Raymond's, and it certainly isn't a pattern to be recommended. That's why
>> it's in the stdlib -- so you don't have to write such code yourself. Same
>> for enums. Yes, it *can* be done. But it takes superhuman skills to get it
>> right and it still won't be maintainable (TBH, every time I see the
>> namedtuple implementation I have to resist the urge to rewrite it. :-)
>
> Heh :-)
>
> I was curious to see how much code in namedtuple actually needed to be
> run through exec, so I forked Raymond's recipe on ActiveState, and
> re-wrote with the Inner class as a regular nested class, and only a
> two-line __new__ method needing to be run through exec.
>
> http://code.activestate.com/recipes/578918-yet-another-namedtuple/
>
> It passes the minimal tests in the recipe. I haven't taken it any
> further.

One nice thing about the current impl is you can actually dump the
full source code of the underlying class implementation. That ties in
to another nice property, which is that anyone that understands class
definitions and exec could maintain most of the current
implementation. It would be tough to write it from scratch (since
there are some subtle details to get right), but it's still a much
lower knowledge barrier to get over than is the case with the
corresponding dynamic type creation code.

By contrast, writing it out directly means reimplementing several
things the interpreter will otherwise take care of, which also means
you may not automatically benefit from changes to the way class
definitions work. For example, that recipe version doesn't appear to
adjust __qualname__ at all, while namedtuple (without any changes to
the namedtuple implementation) automatically sets it to match
__name__.

Maintainability is a complicated thing, and our instincts don't always
get the trade-offs right :)

Cheers,
Nick.

P.S. Despite everything I said above, in most cases, dynamic type
creation is still going to be a better idea. namedtuple itself just
isn't one of those cases.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia