[Python-ideas] Module __getattr__ [Was: breaking out of module execution]

[Jim Jewett]

On Wed, Apr 25, 2012 at 2:35 PM, Matt Joiner <anac...@gmail.com> wrote:
> If this is to be done I'd like to see all special methods supported. One of
> particular interest to modules is __getattr__...

For What It's Worth, supporting __setattr__ and __getattr__ is one of
the few reasons that I have considered subclassing modules.

The workarounds of either offering public set_varX and get_varX
functions, or moving configuration to a separate singleton, just feel
kludgy.

Since those module methods would be defined at the far left, I don't
think it would mess up understanding any more than they already do on
regular classes. (There is always *some* surprise, just because they
are unusual.)

That said, I personally tend to view modules as a special case of
classes, so I wouldn't be shocked if others found it more confusing
than I would -- particularly as to whether or not the module's
__getattr__ would somehow affect the lookup chain for classes defined
within the module.

-jJ
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas

[Gregory P. Smith]

On Fri, Apr 27, 2012 at 6:31 AM, Jim Jewett <jimjj...@gmail.com> wrote:

On Wed, Apr 25, 2012 at 2:35 PM, Matt Joiner <anac...@gmail.com> wrote:
> If this is to be done I'd like to see all special methods supported. One of
> particular interest to modules is __getattr__...

For What It's Worth, supporting __setattr__ and __getattr__ is one of
the few reasons that I have considered subclassing modules.

The workarounds of either offering public set_varX and get_varX
functions, or moving configuration to a separate singleton, just feel
kludgy.

Since those module methods would be defined at the far left, I don't
think it would mess up understanding any more than they already do on
regular classes.  (There is always *some* surprise, just because they
are unusual.)

That said, I personally tend to view modules as a special case of
classes, so I wouldn't be shocked if others found it more confusing
than I would -- particularly as to whether or not the module's
__getattr__ would somehow affect the lookup chain for classes defined
within the module.

-jJ

Making modules "simply" be a class that could be subclasses rather than their own thing _would_ be nice for one particular project I've worked on where the project including APIs and basic implementations were open source but which allowed for site specific code to override many/most of those base implementations as a way of customizing it for your own specific (non open source) environment.  Any APIs that were unfortunately defined as a module with a bunch of functions in it was a real pain to make site specific overrides for.  Anything APIs that were thankfully defined as a class within a module even when there wasn't a real need for a class was much easier to make site specific.

But this is not an easy thing to do.  I wouldn't want functions in a module to need to be declared as class methods with a cls parameter nor would I want an implicit named equivalent of cls; or does that already exist through an existing __name__ style variable today that I've been ignoring?  This could leads to basically treating a module globals() dict as the class dict which at first glance seems surprising but I'd have to ponder this more.

(and yes, note that I am thinking of a module as a class, not an instance)

-gps

[Steven D'Aprano]

Gregory P. Smith wrote:

> Making modules "simply" be a class that could be subclasses rather than
> their own thing _would_ be nice for one particular project I've worked on
> where the project including APIs and basic implementations were open source
> but which allowed for site specific code to override many/most of those
> base implementations as a way of customizing it for your own specific (non
> open source) environment.

This makes no sense to me. What does the *licence* of a project have to do
with the library API? I mean, yes, you could do such a thing, but surely you
shouldn't. That would be like saying that the accelerator pedal should be on
the right in cars you buy outright, but on the left for cars you get on
hire-purchase.

Nevertheless, I think your focus here is on the wrong thing. It seems to me
that you are jumping to an implementation, namely that modules should stop
being instances of a type and become classes, without having a clear idea of
your functional requirements.

The functional requirements *might* be:

"There ought to be an easy way to customize the behaviour of attribute access
in modules."

Or perhaps:

"There ought to be an easy way for one module to shadow another module with
the same name, but still inherit behaviour from the shadowed module."

neither of which *require* modules to become classes.

Or perhaps it is something else... it is unclear to me exactly what problems
you and Jim wish to solve, or whether they're the same kind of problem, which
is why I say the functional requirements are unclear.

Changing modules from an instance of ModuleType to "a class that could be a
subclass" is surely going to break code. Somewhere, someone is relying on the
fact that modules are not types and you're going to break their application.

> Any APIs that were unfortunately defined as a
> module with a bunch of functions in it was a real pain to make site
> specific overrides for.

It shouldn't be. Just ensure the site-specific override module comes first in
the path, and "import module" will pick up the override module instead of the
standard one. This is a simple exercise in shadowing modules.

Of course, this implies that the override module has to override *everything*.
There's currently no simple way for the shadowing module to inherit
functionality from the shadowed module. You can probably hack something
together, but it would be a PITA.



--
Steven

[Guido van Rossum]

On Mon, Apr 30, 2012 at 5:47 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> Gregory P. Smith wrote:
>
>> Making modules "simply" be a class that could be subclasses rather than
>> their own thing _would_ be nice for one particular project I've worked on
>> where the project including APIs and basic implementations were open
>> source
>> but which allowed for site specific code to override many/most of those
>> base implementations as a way of customizing it for your own specific (non
>> open source) environment.

> This makes no sense to me. What does the *licence* of a project have to do
> with the library API? I mean, yes, you could do such a thing, but surely you
> shouldn't. That would be like saying that the accelerator pedal should be on
> the right in cars you buy outright, but on the left for cars you get on
> hire-purchase.

That's an irrelevant, surprising and unfair criticism of Greg's
message. He just tried to give a specific example without being too
specific.

If there is a bunch of functions and you want to replace a few of
those, you can probably get the desired effect quite easily:

from base_module import * # Or the specific set of functions that
comprise the API.

def funct1(<args>): <my version>
def funct2(<args>): <my version>

Not that I would recommend this -- it's easy to get confused if there
are more than a very small number of functions. Also if
base_module.funct3 were to call func2, it wouldn't call the overridden
version.

But all attempts to view modules as classes or instances have lead to
negative results. (I'm sure I've thought about it at various times in
the past.)

I think the reason is that a module at best acts as a class where
every method is a *static* method, but implicitly so. Ad we all know
how limited static methods are. (They're basically an accident -- back
in the Python 2.2 days when I was inventing new-style classes and
descriptors, I meant to implement class methods but at first I didn't
understand them and accidentally implemented static methods first.
Then it was too late to remove them and only provide class methods.)

There is actually a hack that is occasionally used and recommended: a
module can define a class with the desired functionality, and then at
the end, replace itself in sys.modules with an instance of that class
(or with the class, if you insist, but that's generally less useful).
E.g.:

# module foo.py

import sys

class Foo:
def funct1(self, <args>): <code>
def funct2(self, <args>): <code>

sys.modules[__name__] = Foo()

This works because the import machinery is actively enabling this
hack, and as its final step pulls the actual module out of
sys.modules, after loading it. (This is no accident. The hack was
proposed long ago and we decided we liked enough to support it in the
import machinery.)

You can easily override __getattr__ / __getattribute__ / __setattr__
this way. It also makes "subclassing" the module a little easier
(although accessing the class to be used as a base class is a little
tricky: you'd have to use foo.__class__). But of course the kind of
API that Greg was griping about would never be implemented this way,
so that's fairly useless. And if you were designing a module as an
inheritable class right from the start you're much better off just
using a class instead of the above hack.

But all in all I don't think there's a great future in stock for the
idea of allowing modules to be "subclassed". In the vast, vast
majority of cases it's better to clearly have a separation between
modules, which provide no inheritance and no instantiation, and
classes, which provide both. I think Python is better off this way
than Java, where all you have is classes (its packages cannot contain
anything except class definitions).

--
--Guido van Rossum (python.org/~guido)

[Nick Coghlan]

On Tue, May 1, 2012 at 12:33 PM, Guido van Rossum <gu...@python.org> wrote:
> But all in all I don't think there's a great future in stock for the
> idea of allowing modules to be "subclassed". In the vast, vast
> majority of cases it's better to clearly have a separation between
> modules, which provide no inheritance and no instantiation, and
> classes, which provide both. I think Python is better off this way
> than Java, where all you have is classes (its packages cannot contain
> anything except class definitions).

FWIW, in 3.3 the full import machinery will be exposed in
sys.meta_path (and sys.path_hooks), so third parties will be free to
experiment with whatever crazy things they want without having to work
around the implicit import behaviour :)

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Gregory P. Smith]

On Mon, Apr 30, 2012 at 7:33 PM, Guido van Rossum <gu...@python.org> wrote:

On Mon, Apr 30, 2012 at 5:47 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> Gregory P. Smith wrote:
>
>> Making modules "simply" be a class that could be subclasses rather than
>> their own thing _would_ be nice for one particular project I've worked on
>> where the project including APIs and basic implementations were open
>> source
>> but which allowed for site specific code to override many/most of those
>> base implementations as a way of customizing it for your own specific (non
>> open source) environment.

> This makes no sense to me. What does the *licence* of a project have to do
> with the library API? I mean, yes, you could do such a thing, but surely you
> shouldn't. That would be like saying that the accelerator pedal should be on
> the right in cars you buy outright, but on the left for cars you get on
> hire-purchase.

That's an irrelevant, surprising and unfair criticism of Greg's
message. He just tried to give a specific example without being too
specific.

heh. right. I didn't want to drag people into a project that shall not be named because I dislike its design to begin with... I was just suggesting a case where we _would_ have found it useful to treat a module as a class that could be subclassed.  A better overall design pattern using classes in the code's public APIs to start with would have prevented all such need.

I am not strongly in favor of modules being classes but it is an interesting thing to ponder from time to time.

This "oops, I implemented static methods" is a wonderful bit of history! :)