Consider allowing the use of abstractmethod without metaclasses

[Neil Girdhar]

I want to use abstractmethod, but I have my own metaclasses and I don't want to build composite metaclasses using abc.ABCMeta.

Thanks to PEP 487, one approach is to facator out the abstractmethod checks from ABCMeta into a regular (non-meta) class.  So, my first suggestion is to split abc.ABC into two pieces, a parent regular class with metaclass "type":

class AbstractBaseClass:

    def __init_subclass__(cls):

        # Compute set of abstract method names

        abstracts = {name

                     for name, value in vars(cls).items()

                     if getattr(value, "__isabstractmethod__", False)}

        for base in cls.__bases__:

            for name in getattr(base, "__abstractmethods__", set()):

                value = getattr(cls, name, None)

                if getattr(value, "__isabstractmethod__", False):

                    abstracts.add(name)

        cls.__abstractmethods__ = frozenset(abstracts)

My alternative suggestion is to move this logic directly into "type" so that all classes have this logic, and then move abstractmethod into builtins.

Of course, this isn't pressing since I can do this in my own code, it's just a suggestion from a neatness standpoint.

Best,

Neil

[Guido van Rossum]

I suppose it makes sense to allow using `@abstractmethod` without `metaclass=ABCMeta` if we can swing it. In fact, if you're checking your code statically with mypy, that already works: mypy will check for methods remaining abstract regardless of whether that metaclass is used, and the decorator is just not effective at runtime, but it does no harm either.

If you want to pursue an architecture that actually checks for abstract-remaining methods at runtime without using a metaclass, feel free to start working on a PR (or someone else might try this). However, one constraint for such a PR being accepted is that there should absolutely no slowdown for instantiation (in mypy, we found a pretty severe slowdown due to the metaclass).

FWIW, I haven't reviewed it carefully, but I have my doubts about the snippet you showed (it should probably descend down base classes recursively, or, preferably, iterate over the MRO).

--Guido

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--

--Guido van Rossum (python.org/~guido)

[INADA Naoki]

Hi, Neil.

> I want to use abstractmethod, but I have my own metaclasses and I don't want
> to build composite metaclasses using abc.ABCMeta.
>
> Thanks to PEP 487, one approach is to facator out the abstractmethod checks
> from ABCMeta into a regular (non-meta) class. So, my first suggestion is to
> split abc.ABC into two pieces, a parent regular class with metaclass "type":
>

I'm +1 with your idea in performance point of view.

Some people having other language background (C# or Java)
want to use ABC like Java's interface.

But ABC is too heavy to use only for checking abstract methods.
It uses three inefficient WeakSet [1] and it overrides isinstance and issubclass
with slow Python implementation.

[1] WeakSet is implemented in Python, having one __dict__, list and two sets.

And C implementation of gathering abstract methods will reduce
Python startup time too.

## in abc.py

# Helper function. ABCMeta use this too.
# And Python 3.7 can have C implementation of this.

def _init_abstractclass(cls):

# Compute set of abstract method names
abstracts = {name
for name, value in vars(cls).items()
if getattr(value, "__isabstractmethod__", False)}
for base in cls.__bases__:
for name in getattr(base, "__abstractmethods__", set()):
value = getattr(cls, name, None)
if getattr(value, "__isabstractmethod__", False):
abstracts.add(name)
cls.__abstractmethods__ = frozenset(abstracts)

class Abstract:
__init_subclass__ = _init_abstractclass


## usage

import abc

# by decorator
@abc.abstractclass
class AbstractFoo:
@abc.abstractmethod
def foo(self): ...

# by inheritance
class AbstractBar(abc.Abstract):
@abc.abstractmethod
def bar(self): ...


Bests,

INADA Naoki <songof...@gmail.com>

[INADA Naoki]

Hi, Neil.

> I want to use abstractmethod, but I have my own metaclasses and I don't want
> to build composite metaclasses using abc.ABCMeta.
>
> Thanks to PEP 487, one approach is to facator out the abstractmethod checks
> from ABCMeta into a regular (non-meta) class. So, my first suggestion is to
> split abc.ABC into two pieces, a parent regular class with metaclass "type":
>

I'm +1 with your idea in performance point of view.

Some people having other language background (C# or Java)
want to use ABC like Java's interface.

But ABC is too heavy to use only for checking abstract methods.
It uses three inefficient WeakSet [1] and it overrides isinstance and issubclass
with slow Python implementation.

[1] WeakSet is implemented in Python, having one __dict__, list and two sets.

And C implementation of gathering abstract methods will reduce

Python startup time too. Because `_collections_abc` module has many ABCs and
`os` module import it.

## in abc.py

# Helper function. ABCMeta use this too.
# And Python 3.7 can have C implementation of this.

def _init_abstractclass(cls, bases=None):

# Compute set of abstract method names

if bases is None:
bases = cls.__bases__

abstracts = {name
for name, value in vars(cls).items()
if getattr(value, "__isabstractmethod__", False)}

for base in bases:

for name in getattr(base, "__abstractmethods__", set()):
value = getattr(cls, name, None)
if getattr(value, "__isabstractmethod__", False):
abstracts.add(name)
cls.__abstractmethods__ = frozenset(abstracts)


class Abstract:
__init_subclass__ = _init_abstractclass


## usage

import abc

class AbstractBar(abc.Abstract):
@abc.abstractmethod
def bar(self): ...


Bests,

INADA Naoki <songof...@gmail.com>

[Victor Stinner]

Le 20 juil. 2017 3:49 AM, "INADA Naoki" <songof...@gmail.com> a écrit :

> I'm +1 with your idea in performance point of view.

(...)

But ABC is too heavy to use only for checking abstract methods.
It uses three inefficient WeakSet [1] and it overrides isinstance and issubclass
with slow Python implementation.

I don't think that we can get ride of abc from the io and importlib. They are here to stay. I hear your performance analysis.

Why not making abc faster instead of trying to workaround abc for perf issue?

Someone already wrote WeakrefSet, a PR is waiting for our review!

Victor

[INADA Naoki]

Hi, Victor.

> Why not making abc faster instead of trying to workaround abc for perf
> issue?

Current ABC provides:

a) Prohibit instantiating without implement abstract methods.
b) registry based subclassing

People want Java's interface only wants (a). (b) is unwanted side effect.

Additionally, even if CPython provide C implementation of ABCMeta,
other Python implementations won't.
So Abstract Class (not ABC) may be nice on such implementations too.

I'm +1 to implement abc module in C.
And I think (a) can be nice first step, instead of implement all at once.

Regards,

[Neil Girdhar]

Good discussion so far.  Please let me know if I can help with implementation or documentation.

On Thu, Jul 20, 2017 at 8:40 AM INADA Naoki <songof...@gmail.com> wrote:

Hi, Victor.

> Why not making abc faster instead of trying to workaround abc for perf
> issue?

Current ABC provides:

a) Prohibit instantiating without implement abstract methods.
b) registry based subclassing

People want Java's interface only wants (a).  (b) is unwanted side effect.

Right.  (b) is only unwanted because it requires metaclasses, and metaclasses put constraints on inheritance.  

I have switched to using the "AbstractBaseClass" class I defined above in my own code.  If that is the sort of solution that will be undertaken, then there is a question of what to call this class.

If instead, every class will get this functionality automatically (which is more elegant), then someone needs to show that performance is unaffected.

Also, this whole thing might not be that important if (as Guido implies) linters supplant this functionality.  Although, linters would not catch the rare case where classes are programatically-generated.

Additionally, even if CPython provide C implementation of ABCMeta,
other Python implementations won't.
So Abstract Class (not ABC) may be nice on such implementations too.

I'm +1 to implement abc module in C.
And I think (a) can be nice first step, instead of implement all at once.

Regards,

INADA Naoki  <songof...@gmail.com>
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--

---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/r2YLrIEQlig/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ivan Levkivskyi]

To be honest, I am not very happy with addition of a new special class. Imagine that the PEP 544 will be accepted (and I hope so).

Then we would have, abstract classes, abstract base classes, and protocols. I think users will be overwhelmed by having
three similar concepts instead of one.

I think we still could squeeze a lot of performance from good old ABCs by optimizing various parts and reimplementing some parts in C.

In fact, my desire to optimize and rewrite ABCMeta in C is partially due to reluctance to add yet another concept of "abstractness".

--

Ivan

[INADA Naoki]

On Fri, Jul 21, 2017 at 12:12 AM, Ivan Levkivskyi <levki...@gmail.com> wrote:
> To be honest, I am not very happy with addition of a new special class.
> Imagine that the PEP 544 will be accepted (and I hope so).
> Then we would have, abstract classes, abstract base classes, and protocols.
> I think users will be overwhelmed by having
> three similar concepts instead of one.

Hmm, couldn't split protocol and ABC?

Of course, existing ABCs should be ABC for backward compatibility.
But any reason to force using ABCMeta for user defined classes?

I hate subclassing ABC because concrete classes which mix-in some ABC
are forced to use it.

>
> I think we still could squeeze a lot of performance from good old ABCs by
> optimizing various parts and reimplementing some parts in C.
> In fact, my desire to optimize and rewrite ABCMeta in C is partially due to
> reluctance to add yet another concept of "abstractness".
>

Even if it's implemented in C, issubclass implementation is much complicated
than normal type.
I don't want to introduce unnecessary complexity because I'm minimalist.

Regards,

> --
> Ivan

[Ivan Levkivskyi]

On 20 July 2017 at 19:51, INADA Naoki <songof...@gmail.com> wrote:

On Fri, Jul 21, 2017 at 12:12 AM, Ivan Levkivskyi <levki...@gmail.com> wrote:
> To be honest, I am not very happy with addition of a new special class.
> Imagine that the PEP 544 will be accepted (and I hope so).
> Then we would have, abstract classes, abstract base classes, and protocols.
> I think users will be overwhelmed by having
> three similar concepts instead of one.

Hmm, couldn't split protocol and ABC?

Unfortunately no, it was considered and rejected for various reasons (in particular to provide smooth transition to protocols).

 

> I think we still could squeeze a lot of performance from good old ABCs by
> optimizing various parts and reimplementing some parts in C.
> In fact, my desire to optimize and rewrite ABCMeta in C is partially due to
> reluctance to add yet another concept of "abstractness".
>

Even if it's implemented in C, issubclass implementation is much complicated
than normal type.
I don't want to introduce unnecessary complexity because I'm minimalist.

This complexity is already there, and attempt to reduce might lead to actually an increase of complexity.

This is probably the case where I would be with Raymond in terms of performance vs ease of maintenance.

--

Ivan

[Nathaniel Smith]

On Jul 20, 2017 05:39, "INADA Naoki" <songof...@gmail.com> wrote:

Hi, Victor.

> Why not making abc faster instead of trying to workaround abc for perf
> issue?

Current ABC provides:

a) Prohibit instantiating without implement abstract methods.
b) registry based subclassing

People want Java's interface only wants (a).  (b) is unwanted side effect.

Except (b) is what allows you to subclass an ABC without using the ABC metaclass :-)

I wonder if it would make sense to go further and merge *both* of these features into regular classes.

Checking for @abstractmethod in type.__new__ surely can't be that expensive, can it?

And if regular types supported 'register', then it would allow for a potentially simpler and faster implementation. Right now, superclass.register(subclass) has to work by mutating superclass, because that's the special ABCMeta object, and this leads to complicated stuff with weakrefs and all that. But if this kind of nominal inheritance was a basic feature of 'type' itself, then it could work by doing something like

  subclass.__nominal_bases__ += (superclass,)

and then precalculating the "nominal mro" just like it already precalculates the mro, so issubclass/isinstance would remain fast.

I guess enabling this across the board might cause problems for C classes whose users currently use isinstance to get information about the internal memory layout.

-n

[INADA Naoki]

INADA Naoki <songof...@gmail.com>

On Fri, Jul 21, 2017 at 2:59 AM, Ivan Levkivskyi <levki...@gmail.com> wrote:
> On 20 July 2017 at 19:51, INADA Naoki <songof...@gmail.com> wrote:
>>
>> On Fri, Jul 21, 2017 at 12:12 AM, Ivan Levkivskyi <levki...@gmail.com>
>> wrote:
>> > To be honest, I am not very happy with addition of a new special class.
>> > Imagine that the PEP 544 will be accepted (and I hope so).
>> > Then we would have, abstract classes, abstract base classes, and
>> > protocols.
>> > I think users will be overwhelmed by having
>> > three similar concepts instead of one.
>>
>> Hmm, couldn't split protocol and ABC?
>>
>
> Unfortunately no, it was considered and rejected for various reasons (in
> particular to provide smooth transition to protocols).
>

Sorry about my poor English.
"split" meant "optionally ABC".

I understand that existing classes (like typing.Sequence) must be ABC.
But why new user defined protocol must be ABC?

>> > by
>> > optimizing various parts and reimplementing some parts in C.
>> > In fact, my desire to optimize and rewrite ABCMeta in C is partially due
>> > to
>> > reluctance to add yet another concept of "abstractness".
>> >
>>
>> Even if it's implemented in C, issubclass implementation is much
>> complicated
>> than normal type.
>> I don't want to introduce unnecessary complexity because I'm minimalist.
>>
>
> This complexity is already there, and attempt to reduce might lead to
> actually an increase of complexity.
> This is probably the case where I would be with Raymond in terms of
> performance vs ease of maintenance.
>

Sorry again. I meant I don't want import the complexity to my class
when I don't need it.
In other words, I hate inheriting ABC when I don't need register based subclass.


> --
> Ivan
>
>

[INADA Naoki]

>
> I wonder if it would make sense to go further and merge *both* of these
> features into regular classes.
>
> Checking for @abstractmethod in type.__new__ surely can't be that expensive,
> can it?
>

But it affects startup time.
It iterate all of the namespace and tries `getattr(obj,
`__isabstractmethod__`, False).
It means massive AttributeErrors are raised and cleared while loading
large library.

OTOH, I have another idea:

class AbstractFoo:
def foo(self):
...
__abstractmethods__ = ("foo",)

In this idea, `type.__new__` can check only `__abstractmethods__`.

> And if regular types supported 'register', then it would allow for a
> potentially simpler and faster implementation. Right now,
> superclass.register(subclass) has to work by mutating superclass, because
> that's the special ABCMeta object, and this leads to complicated stuff with
> weakrefs and all that. But if this kind of nominal inheritance was a basic
> feature of 'type' itself, then it could work by doing something like
>
> subclass.__nominal_bases__ += (superclass,)
>
> and then precalculating the "nominal mro" just like it already precalculates
> the mro, so issubclass/isinstance would remain fast.
>

I don't like it.
In 99.9% of my classes, I don't need register based subclassing.

> I guess enabling this across the board might cause problems for C classes
> whose users currently use isinstance to get information about the internal
> memory layout.
>
> -n