[Python-Dev] Proto-PEP part 1: Forward declaration of classes

1,589 views
Skip to first unread message

Larry Hastings

unread,
Apr 22, 2022, 9:19:01 PM4/22/22
to Python-Dev


This document is a loose proto-PEP for a new "forward class" / "continue
class" syntax.  Keep in mind, the formatting is a mess. If I wind up
submitting it as a real PEP I'll be sure to clean it up first.


/arry

--------------------------------------


PEP XXXX: Forward declaration of classes

Overview
--------

Python currently has one statement to define a class, the `class` statement:

```Python
    class X():
        # class body goes here
        def __init__(self, key):
            self.key = key
```

This single statement declares the class, including its bases and metaclass,
and also defines the contents of the class in the "class body".

This PEP proposes an additional syntax for declaring a class which splits
this work across two statements:
* The first statement is `forward class`, which declares the class and binds
  the class object.
* The second statement is `continue class`, which defines the contents
  of the class in the "class body".

To be clear: `forward class` creates the official, actual class object.
Code that wants to take a reference to the class object may take references
to the `forward class` declared class, and interact with it as normal.
However, a class created by `forward class` can't be *instantiated*
until after the matching `continue class` statement finishes.

Defining class `X` from the previous example using this new syntax would
read
as follows:

```
    forward class X()

    continue class X:
        # class body goes here
        def __init__(self, key):
            self.key = key
```

This PEP does not propose altering or removing the traditional `class`
statement;
it would continue to work as before.


Rationale
---------

Python programmers have had a minor problem with classes for years: there's
no way to have early-bound circular dependencies between objects. If A
depends on B, and B depends on A, there's no linear order that allows
you to cleanly declare both.

Most of the time, the dependencies were in late-binding code, e.g. A refers
to B inside a method.  So this was rarely an actual problem at runtime. 
When
this problem did arise, in code run at definition-time, it was usually only
a minor headache and could be easily worked around.

But the explosion of static type analysis in Python, particularly with
the `typing` module and the `mypy` tool, has made circular definition-time
dependencies between classes commonplace--and much harder to solve.  Here's
one simple example:

```Python
    class A:
        value: B

    class B:
        value: A
```

An attribute of `B` is defined using a type annotation of `A`, and an
attribute of `A` is defined using a type annotation of `B`. There's
no order to these two definitions that works; either `A` isn't defined
yet, or `B` isn't defined yet.

Various workarounds and solutions have been proposed to solve this problem,
including two PEPs: PEP 563 (automatic stringized annotations) and PEP 649
(delayed evaluation of annotations using functions).
But nothing so far has been both satisfying and complete; either it
is wordy and clumsy to use (manually stringizing annotations), or it
added restrictions and caused massive code breakage for runtime use of
annotations (PEP 563), or simply didn't solve every problem (PEP 649).
This proposed  `forward class` / `continue class` syntax should permit
solving *every* forward-reference and circular-reference problem faced
in Python, using an elegant and Pythonic new syntax.

As a side benefit, `forward class` and `continue class` syntax enables
rudimentary separation of "interface" from "implementation", at least for
classes.  A user seeking to "hide" the implementation details of their
code could put their class definitions in one module, and the
implementations of those classes in a different module.

This new syntax is not intended to replace the traditional `class`
declaration syntax in Python.  If this PEP were accepted, the `class`
statement would still be the preferred mechanism for creating classes
in Python; `forward class` should only be used when it confers some
specific benefit.


Syntax
------

The `forward class` statement is the same as the `class` statement,
except it doesn't end with a colon and is not followed by an indented block.
Without any base classes or metaclass, the `forward class` statement is
as follows:

```
    forward class X
```

This would declare class `X`.

If `X` needs base classes or metaclass, the corresponding `forward
class` statement
would be as follows, rendered in a sort of "function prototype" manner:

```
    forward class X(*bases, metaclass=object, **kwargs)
```

The `continue class` statement is similar to a `class` statement
without any bases or metaclass.  It ends with a colon,
and is followed by the "class body":

    continue class X:
        # class body goes here
        pass

One important difference: the `X` in `continue class X:` is not a *name*,
it's an *expression*.  This code is valid:

```
    forward class X()
    snodgrass = X

    continue class snodgrass:
        # class body goes here
        pass
```

as well as this:

```
    import my_module

    continue class my_module.X:
        # class body goes here
        pass
```

Using this new syntax, the forward-reference problem illustrated
in the *Rationale* section above is now easy to solve:

```Python
    forward class B

    class A:
        value: B = None

    continue class B:
        value: A = None
```

One final note.  Why must the base and metaclass be declared
with the `forward class` statement?  The point of this new
syntax is to allow creating the real class object, permitting
users of the class to take references to it early, before it's
fully defined.  And the class could be declared with a
metaclass, and the metaclass could have a `__new__`, which
means it's responsible for creating the class object, and
this syntax would have to preserve that behavior.

(This small detail is about to complicate this proposal a great
deal!)


#### Semantics of forward-declared class objects

`forward class X` declares a class, but the class is explicitly
not yet fully defined.  It won't be fully defined and ready to
be instantiated until after the corresponding `continue class`
statement.  We'll refer to a class object in this state as
a "forward-declared class object".  How does this object behave?

As per the "consenting adults" rule, the forward-declared class
object must permit most operations.  You should be able to examine
the object, compare it to other objects, inspect some attributes
(`__name__`, `__mro__`, `__class__`), and even set attributes.

However, the user isn't permitted to instantiate a forward-declared
class object until after the corresponding `continue class X`.
We ensure this with a new dunder attribute, `__forward__`,
which if present tells the Python runtime that this is a
forward-declared class object.  The `continue class` statement
would delete this attribute from the object, after which it
could be instantiated.

(Users could work around this constraint, or even delete `__forward__`
if they so chose--again, the "consenting adults" rule applies.)

It's explicitly permissible to create a forward-declared class
object that never gets finished with a `continue class` statement.
If all you need is an object that represents the class--say,
to satisfy static type annotation use cases--a forward-declared
class object works fine.

A subsequent section will address the complexities of
how `forward class` and `continue class` interact with metaclasses.
For now, a note about forward-declared class objects declared with
a metaclass implementing `__prepare__`.  The forward-declared class
object *dict* will be the "dict-like object" returned by the
`metaclass.__prepare__()` method.  This "dict-like object" won't
be processed and discarded until after `continue class` processes
the class body and calls the appropriate methods in the metaclass.


#### Semantics of `continue class`

`continue class` may only be run on a class once.
(As Eric V. Smith pointed out in response to an early version of
this proposal, allowing multiple "continue" declarations on the
same class would lead directly to language-condoned monkey-patching.)


#### Decorators

Both the `forward class` and `continue class` statements
support decorators, and the user may use decorators with either
or both statements for the same class.  But now that we've
split the responsibilities of the `class` statement between
these two new statements, which decorator goes with which
statement becomes a novel concern.  In general, decorators
that don't examine the contents of the class, but simply
want to register the class object and its name, can decorate
the `forward class` statement.  Also, class decorators that
want to return a different object for the class should decorate
`forward class`.  But all decorators that meaningfully examine
the contents of the class should decorate the `continue class`
statement.

Unfortunately, there are some decorators that can't work properly
with either `forward class` *or* `continue class`: a decorator
that meaningfully examine the declared contents of that class, but
also return an object other than the original class passed in.
In that case, the user cannot declare this class with
`forward class`; they must declare it with the conventional `class`
statement.

#### __slots__

This leads us to an example of a decorator that, as of 3.10,
wouldn't be usable with classes declared by `forward class`.
It's the new 3.10 feature `@dataclass(slots=True)`.  When called
in this way, dataclass examines the attributes of the class it
has decorated, dynamically constructs a new class using `__slots__`,
and returns this new class.

Since this decorator meaningfully examines the class, it must
be used with `continue class`.  But, this decorator also returns
an object other the original class, which means it's inapproriate
for `continue class` and should be called with `forward class`.
What to do?

We have a separate idea to ameliorate this specific situation.
Right now, a class that uses `__slots__` *must* define them in
the class body, as that member is read before the class name is
bound (or before any descriptors are run).  But we can simply
relax that, and make processing `__slots__` lazy, so that it
isn't examined until the first time the class is *instantiated.*
This would mean `@dataclass(slots=True)` could simply return the
original class, and thus would work fine when decorating a
`continue class` statement.


#### Metaclasses

The existing semantics of metaclasses present a thorny problem
for `forward class` and `continue class`.  First, a review of
how class definition works.

Most of the mechanisms involved with defining a class happen
internally to the interpreter.  However, there are a number
of special methods (aka "dunder methods") called during class
construction that are overridable by user code.  Empirical testing
with the current version of Python (as of this writing, 3.10.4)
reveals the order in which all this work is done.

When Python executes the definition of this class:

```Python
    class Foo(BaseClass, metaclass=MetaClass):
        # class body is here
        pass
```

these events are visible in this order:

1. Python calls `MetaClass.__prepare__`.
2. Python executes the "class body" for class Foo.
3. Python calls `MetaClass.__new__`.
4. Python calls `BaseClass.__init_subclass__`.
   (If multiple base classes define `__init_subclass__`,
   they're called in MRO order.)
5. Python calls `MetaClass.__init__`.
6. The `class` statement binds the class object to the name `Foo`.

The big problem this presents for `forward class`: the
"class body" is executed before the `MetaClass.__new__`.
This is necessary because one of the parameters to `MetaClass.__new__`
is `namespace`, the "dict-like object" returned by `MetaClass.__prepare__`
and initialized by executing the class body using that object
as a sort of locals dictionary.

This creates a chicken-and-egg problem: `forward class` needs
to define the class object, but the class object is defined
by `MetaClass.__new__`, and `MetaClass.__new__` can't run until
after the class body, which we don't run until the `continue class`
statement, which must be after `forward class`.

The unfortunate but necessary solution: split `__new__` into
two new special methods on metaclasses, `__new_forward__`
and `__new_continue__`.  As a reminder, here's the prototype
for `__new__` on a metaclass:

```Python
    def __new__(metaclass, name, bases, namespace, **kwargs):
```

The two new special methods would have the following prototypes:

```Python
    def __new_forward__(metaclass, name, bases, namespace, **kwargs):

    def __new_continue__(metaclass, cls, **kwargs):
```

`__new_forward__` creates the class object.  It sets the `namespace`
member as the class dict, but in general should not examine it
contents.  (Specifically, `__new_forward__` cannot make any assumptions
about whether or not the class body has been executed yet; more on this
in a moment.)

`__new_continue__` is guaranteed to be called after the
class body has been executed.

The externally-visible parts of class construction would
run in a different order for classes constructed using
`forward class` and `continue class`.  First, the visible
interactions from the `forward class` statement:

1. Python calls `MetaClass.__prepare__`.
2. Python calls `MetaClass.__new_forward__`.
3. The `forward class` statement binds the (forward-declared)
   class object to the name `Foo`.

And here are the visible interactions from the
`continue class` statement:

1. Python executes the class body.
2. Python calls `MetaClass.__new_continue__`.
3. Python calls `BaseClass.__init_subclass__`.
   (If multiple base classes define `__init_subclass__`,
   they're called in MRO order.)
4. Python calls `MetaClass.__init__`.

It's important to note that, while `namespace` is passed
in to `__new_forward__`, it's not yet initialized with the
class body.  It's passed in here because the "dict-like object"
returned by `MetaClass.__prepare__` is used as the `__dict__`
for the forward-declared class object.

(This is also novel.  Normally the "dict-like object" is
used as the namespace for the class body, then its contents
are copied out and it is discarded.  Here it will also be
used as the `__dict__` for the forward-declared class object
until the `continue class` statement executes.)

Splitting `__new__` into two methods in this manner has several
ramifications for existing code.

First, Python still needs to support `MetaClass.__new__`
for backwards compatibility with existing code.  Therefore,
when executing the `class` statement, Python will still call
`MetaClass.__new__`.  In fact, for maximum backwards
compatibility, the order of externally-visible events
for the `class` statement should not change at all.

The default implementation of `MetaClass.__new__` will be
changed to call `__new_forward__` and `__new_continue__`.
The implementation will be similar to the following
pseudo-code:

```Python
    def __new__(metaclass, name, bases, namespace, **kwargs):
        cls = metaclass.__new_forward__(metaclass, name, bases,
namespace, **kwargs)
        metaclass.__new_continue__(metaclass, cls, namespace, **kwargs)
        return cls
```

This means the order of events will be slightly different
between a class defined with the `class` statement and
a class defined with the `forward class` and `continue class`
statements.  With a `class` statement, the class body will be
run *before* `__new_forward__` is called, but with a `forward class`
statement, the class body will be run *after* `__new_forward__`
is called.  (This is why `__new_forward__` cannot know in advance
whether or not the class body has been called, and the `namespace`
has been filled in.)

User code that defines its own metaclass with its own `__new__`
must also continue to work.  But this leads us to a dangerous
boundary condition:
  * if user code defines a metaclass, and
  * that metaclass defines `__new__` but not `__new_forward__` or
    `__new_continue__`, and
  * user code then uses that metaclass in a `forward class`
    declaration, then
Python must throw a `TypeError` exception.  This situation is
unsafe: clearly the intention with the user's metaclass is to
override some behavior in `__new__`, but the `forward new` statement
will never call `__new__`.

(It's safe to use a metaclass with `forward class` if it doesn't
define `__new__`, or if it defines both `__new__` and either
`__new_forward__` or `__new_continue__`.  It's also safe to
use a metaclass with `class` if it defines either `__new_forward__`
or `__new_continue__` but not `__new__`, because the default `__new__`
will call both `__new_forward__` and `__new_continue__`.)

Going forward, best practice for metaclasses would be to only
implement `__new_forward__` and `__new_continue__`.
Code with metaclasses that wanted to simultaneously support
versions of Python with these new dunder methods *and* older
versions of Python that predate this change would likely have
to conditionally define their own `__new__`, best practices
on this approach TBD.

#### Interactions between `class`, `forward class`, and `continue class`

`class` and `forward class` both bind a name to a newly-created object.
Thus, in the same way that you can have two `class` statements that
bind and re-bind the same name:

```Python
    class C:
        pass
    class C:
        pass
```

You can execute `class` and `forward class` statements in any order
to bind and re-bind the same name:

```Python
    class C:
        pass
    forward class C
```

This works as expected; when this code executes, the previous objects
are dereferenced, and only the last definition of `C` is kept.

Executing a `continue class` statement with a class defined by the
`class` statement raises a `ValueError` exception.
Executing a `continue class` statement with a class defined by the
`forward class` statement that has already had `continue class`
executed on it raises a `ValueError` exception.

It's expected that knowledgeable users will be able to trick
Python into executing `continue class` on the same class multiple
times by interfering with "dunder" attributes.  The same tricks
may also permit users to trick Python into executing `continue class`
on a class defined by the `class` statement.  This is undefined
and unsupported behavior, but Python will not prevent it.


Final Notes
-----------

#### Alternate syntax

Instead of `forward class`, we could use `def class`.  It's not as
immediately clear, which is why this PEP prefers `forward class`.
But this alternate syntax has the advantage of not adding a new
keyword to the language.


#### forward classes and PEP 649

I suggest that `forward class` meshes nicely with PEP 649.

PEP 649 solves the forward-reference and circular-reference
problem for a lot of use cases, but not all.  So by itself
it's not quite a complete solution to the problem.

This `forward class` proposal should solve *all* the
forward-reference and circular-reference problems faced
by Python users today.  However, its use requires
backwards-incompatible code changes to user code.

By adding both PEP 649 and `forward class` to Python, we
get the best of both worlds.  PEP 649 should handle most
forward-reference and circular-reference problems, but the
user could resort to `forward class` for the stubborn edge
cases PEP 649 didn't handle.

In particular, combining this PEP with PEP 649 achieves total
coverage of the challenges cited by PEP 563's *Rationale* section:

> * forward references: when a type hint contains names that have not been
>   defined yet, that definition needs to be expressed as a string literal;
> * type hints are executed at module import time, which is not
>   computationally free.

PEP 649 solves many forward reference problems, and delays the evaluation
of annotations until they are used.  This PEP solves the remaining forward
reference problems.


### A proof-of-concept using decorators

I've published a repo with a proof-of-concept of
the `forward class` / `continue class` syntax,
implemented using decorators.
It works surprisingly well, considering.
You can find the repo here:

    https://github.com/larryhastings/forward

Naturally, the syntax using this decorators-based version
can't be quite as clean.  The equivalent declaration for
`class X` using these decorators would be as follows:

```Python
    from forward import *

    @forward()
    class X():
       ...

    @continue_(X)
    class _:
       # class body goes here
       pass
```

Specifically:

* You have to make the `forward` module available somehow.  You can just
copy the
  `forward` directory into the directory you want to experiment in, or
you can
  install it locally in your Python install or venv by installing the
`flit`
  package from PyPI and running `flit install -s` .
* You must import and use the two decorators from the `forward` module.
  The easiest way is with `from forward import *` .
* For the `forward class` statement, you instead decorate a conventional
class
  declaration with `@forward()`.  The class body should be empty, with
either
  a single `pass` statement or a single ellipsis `...` on a line by itself;
  the ellipsis form is preferred.  You should name this class with the
desired
  final name of your class.
* For the `continue class` statement, you instead decorate a conventional
  class declaration with `@continue_()`, passing in the
forward-declared class
  object as a parameter to the decorator.  You can use the original name of
  the class if you wish, or a throwaway name like `_` as per the example.
* You may use additional decorators with either or both of these decorators.
  However it's vital that `@forward()` and `@continue_()` are the
  *first* decorators run--that is, they should be on the *bottom* of the
  stack of decorators.

Notes and caveats on the proof-of-concept:

* The `continue_` decorator returns the original "forwarded" class object.
  This is what permits you to stack additional decorators on the class.
  (But, again, you *must* call the `continue_` decorator first--it should
  be on the bottom.)
* To use `__slots__`, you will have to declare them in the `forward` class.
* The proof-of-concept can't support classes that inherit from a class
  which defines `__init_subclass__`.
* Like the proposed syntax, this proof-of-concept doesn't support
  decorators that both examine the contents of the class *and* return
  a different object, e.g. `@dataclass(slots=True)` in Python 3.10.
* This proof-of-concept doesn't work with metaclasses that
  override either `__new__` or `__init__`, where those functions
  examine the `namespace` argument in any meaningful way.


#### tools/

There are some tools in the `tools/` directory that will (attempt to)
automatically add or remove the `@forward()` decorator to class definitions
in Python scripts.  It turns this:

```Python
    class foo(...):
        pass
```

into this:

```Python
    @forward()
    class foo(...):
        ...

    @continue_(foo)
    class _____:
        pass
```

`tools/edit_file.py` will edit one or more Python files specified on the
command-line, making the above change.  By default it will toggle the
presence of `@forward`
decorators.  You can also specify explicit behavior:

`-a` adds `@forward()` decorators to `class` statements that don't have
them.
`-r` removes `@forward` decorators, changing back to conventional
`class` statements.
`-t` requests that it "toggle" the state of `@forward()` decorators.

The parser is pretty dumb, so don't run it on anything precious. If it
goofs up, sorry!

`tools/edit_tree.py` applies `edit_py.py` to all `*.py` files found
anywhere under
a particular directory.

`tools/edit_stdlib.py` was an attempt to intelligently apply
`edit_file.py` to the `Lib`
tree of a CPython checkout.  Sadly, the experiment didn't really work
out; it seemed
like there were so many exceptions where the brute-force modification
didn't work,
either due to descriptors, metaclasses, or base classes with
`__init_subclass__`,
that I gave up on the time investment.  It's provided here in a
non-functional
state in case anyone wants to experiment with it further.

Also, it's intentionally delicate; it only works on git checkout trees,
and only with
one specific revision id:

    7b87e8af0cb8df0d76e8ab18a9b12affb4526103

#### Postscript

Thanks to Eric V. Smith and Barry Warsaw for proofreading and ideas.
Thanks in particular to Eric V. Smith for the idea about making
`__slots__` processing lazy.
Thanks to Mark Shannon for the idea of prototyping `forward class`
and `continue class` using decorators and simply copying the
attributes.


_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/CW6Z3OS42DYNMO4Y42R6O3LVTPPAA57X/
Code of Conduct: http://python.org/psf/codeofconduct/

Chris Angelico

unread,
Apr 22, 2022, 10:18:30 PM4/22/22
to Python-Dev
On Sat, 23 Apr 2022 at 11:16, Larry Hastings <la...@hastings.org> wrote:
> This PEP proposes an additional syntax for declaring a class which splits
> this work across two statements:
> * The first statement is `forward class`, which declares the class and binds
> the class object.
> * The second statement is `continue class`, which defines the contents
> of the class in the "class body".
>
> [chomp all the details]

Hmm. The "continue class" part looks great IMO (and I have in fact
implemented the decorator version, which is clunky - your version,
where "continue class EXPR:" will reference any class - is way more
flexible than anything a decorator can do easily), but I'm unsure
about the forward class. How is it different from subclassing an ABC?
You're claiming a keyword (and one which is definitely going to
conflict - if this goes forward in this form, I would certainly be
bikeshedding the exact choice of keyword) for something that's
extremely similar to simply defining a class, and then marking it as
abstract.

What happens if you try to continue a non-forward class? For example:

class Demo: pass
continue class Demo: pass

If that works fine, then I'd be inclined to downgrade "forward class"
to a keyword argument (like metaclass is), eg "class
Demo(abstract=True)" or "class Demo(forward=True)".

But, I definitely like the "continue class" part of the idea, and
would love to see that happen. Of course, it's going to be horribly
abused for monkeypatching, but we could do that already anyway :)

ChrisA
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/LDYKP7NSVUECEZPW7SP5FMZLASMGJ6TT/

Terry Reedy

unread,
Apr 22, 2022, 10:37:10 PM4/22/22
to pytho...@python.org
On 4/22/2022 9:13 PM, Larry Hastings wrote:

>     forward class X()

New keywords are a nuisance. And the proposed implementation seems too
complex. How about a 'regular' class statement with a special marker of
some sort.
Example: 'body=None'.

Either __new__ or __init__ could raise XError("Cannot instantiate until
this is continued.", so no special instantiation code would needed and X
could be a real class, with a special limitation.

>     continue class X:
>         # class body goes here
>         def __init__(self, key):
>             self.key = key

'continue' is already a keyword. Given that X is a real class, could
implementation be
X.__dict__.update(new-body-dict)


--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/UAZ5P7N5S4OWFSVZSKK6DUEPZTQPV3GP/

Larry Hastings

unread,
Apr 22, 2022, 10:45:49 PM4/22/22
to pytho...@python.org


On 4/22/22 19:17, Chris Angelico wrote:
I'm unsure about the forward class. How is it different from subclassing an ABC?

They're just different objects.  A subclass of an ABC is either itself another abstract base class, which will never be instantiatable, or a non-abstract class, which is immediately instantiatable.  A forward-declared class object is not currently instantiatable, and is not fully defined, but will become fully defined and instantiatable after the matching "continue class" statement.


What happens if you try to continue a non-forward class?

From the proto-PEP:

Executing a `continue class` statement with a class defined by the `class` statement raises a `ValueError` exception.

And also:

It's expected that knowledgeable users will be able to trick Python into executing `continue class` on the same class multiple times by interfering with "dunder" attributes.  The same tricks may also permit users to trick Python into executing `continue class` on a class defined by the `class` statement.  This is undefined and unsupported behavior, but Python will not prevent it.


/arry

Larry Hastings

unread,
Apr 22, 2022, 11:18:57 PM4/22/22
to pytho...@python.org


On 4/22/22 19:36, Terry Reedy wrote:
On 4/22/2022 9:13 PM, Larry Hastings wrote:
     forward class X()

New keywords are a nuisance.  And the proposed implementation seems too complex.

My proposed implementation seemed necessary to handle the complexity of the problem.  I would welcome a simpler solution that also worked for all the same use cases.


How about a 'regular' class statement with a special marker of some sort.  Example: 'body=None'.

It's plausible.  I take it "body=None" would mean the declaration would not be permitted to have a colon and a class body.  So now we have two forms of the "class" statement, and which syntax you're using is controlled by a named parameter argument.

I think this "body=None" argument changing the syntax of the "class" statement is clumsy.  It lacks the visibility and clarity of "forward class"; the keyword here makes it pretty obvious that this is not a conventional class declaration.  So I still prefer "forward class".

In my PEP I proposed an alternate syntax for "forward class": "def class", which has the feature that it doesn't require adding a new keyword.  But again, I don't think it's as clear as "forward class", and I think clarity is vital here.


Either __new__ or __init__ could raise XError("Cannot instantiate until this is continued.", so no special instantiation code would needed and X could be a real class, with a special limitation.

Yes, my proposal already suggests that __new__ raise an exception.  That's not the hard part of the problem.

The problem with X being a "real class" is that creating a "real class" means running all the class creation code, and a lot of the class creation code is designed with the assumption that the namespace has already been filled by executing the class body.

For example, Enum in enum.py relies on EnumMeta, which defines __new__, which examines the already-initialized namespace of the class you want to create.  If you propose a "body=None" class be a "real class object", then how do you declare a class that inherits from Enum using "body=None"?  Creating the class object will call EnumMeta.__new__, which needs to examine the namespace, which hasn't been initialized yet.

Changing the class object creation code so we can construct a class object in two steps, with the execution of the class body being part of the second step, was the major sticking point--and the source of most of the complexity of my proposal.


     continue class X:
         # class body goes here
         def __init__(self, key):
             self.key = key

'continue' is already a keyword.

I'm aware.  I'm not sure why you mentioned it.


Given that X is a real class, could implementation be X.__dict__.update(new-body-dict)

That's what the proof-of-concept does.  But the proof-of-concept fails with a lot of common use cases:

  • metaclasses that override __new__ or __init__
  • base classes that implement __init_subclass__

because these methods assume the namespace of the class has already been filled in, but it doesn't get filled in until the @continue_() class decorator.  Handling these cases is why my proposal is sadly as complex as it is, and why in practice the proof-of-concept doesn't work a lot of the time.


/arry

Steven D'Aprano

unread,
Apr 23, 2022, 12:05:43 AM4/23/22
to pytho...@python.org
On Fri, Apr 22, 2022 at 06:13:57PM -0700, Larry Hastings wrote:

> This PEP proposes an additional syntax for declaring a class which splits
> this work across two statements:
> * The first statement is `forward class`, which declares the class and binds
>   the class object.
> * The second statement is `continue class`, which defines the contents
>   of the class in the "class body".
>
> To be clear: `forward class` creates the official, actual class object.
> Code that wants to take a reference to the class object may take references
> to the `forward class` declared class, and interact with it as normal.
> However, a class created by `forward class` can't be *instantiated*
> until after the matching `continue class` statement finishes.

Since the "forward class" is a real class, it doesn't need any new
syntax to create it. Just use plain old regular class syntax.

class X(object):
"""Doc string"""
attribute = 42


And now we have our X class, ready to use in annotations.

To add to it, or "continue class" in your terms, we can already do this:

X.value = "Hello World"

Methods are a bit trickier. The status quo pollutes the global
namespace, and repeats the method name three times:

def method(self):
pass

X.method = method

but we can do better with just a small amount of new syntax:

def X.method(self):
pass

This has been proposed before, and rejected for lack of any real need.
But your "forward class/continue class" suggests that the need has now
appeared.

So where you would write this:


    forward class Node()

    continue class Node:
"""Node in a linked list."""
        def __init__(self, value, next:Optional[Node]):
            self.value = value
self.next = next


we could instead do this:

    class Node():
"""Node in a linked list."""

    def Node.__init__(self, value, next:Optional[Node]):
        self.value = value
self.next = next


The only downside -- although perhaps that's an upside -- is that we
save an indent level, which may make it marginally harder to recognise
which function defs are being added into a class, and which are not.

If that really is an issue, then we could keep the "continue class ..."
syntax solely to give us that block structure.

Counter proposal:

`continue class expression:` evaluates expression to an existing class
object (or raises an exception) and introduces a block. The block is
executed inside that class' namespace, as the `class` keyword does,
except the class already exists.

But we don't need the "forward class". Just create a class!

Any methods, docstrings, attributes etc which aren't part of a forward
reference cycle can go in the initial class creation, and only those
which are part of a cycle need to be added afterwards.


Here is your circular class example:


> ```Python
>     class A:
>         value: B
>
>     class B:
>         value: A
> ```

Isn't this case solved by either forward references:

class A:
value: "B"

or by either of PEP 563 or PEP 649? I don't think this is a compelling
example. But let's continue anyway.

That could become:

class A:
pass

class B:
value: A # This is fine, A exists.

A.value: B # And here B exists, so this is fine too.


No new syntax is needed. This is already legal. We only need minimal new
syntax to allow methods to be defined outside of their class body:

def A.method(self):
...


with no need to overload `continue` for a second meaning. Or we could
use the `continue A` syntax, but either way, the `forward A` syntax
seems to be unneeded.


> But nothing so far has been both satisfying and complete; either it
> is wordy and clumsy to use (manually stringizing annotations), or it
> added restrictions and caused massive code breakage for runtime use of
> annotations (PEP 563), or simply didn't solve every problem (PEP 649).

But doesn't PEP 649 solve problems which this proposal does not?
Delaying the evaluation of annotations is more general than merely the
"forward reference" problem.


> This proposed  `forward class` / `continue class` syntax should permit
> solving *every* forward-reference and circular-reference problem faced
> in Python,

I think that's overselling the concept. How about forward references for
function defaults?

def func(arg=func(0)):
if arg == 0:
return "something"
...


Obviously there are other ways to solve that, but the point is, your
forward/continue class proposal is not one of them!



> using an elegant and Pythonic new syntax.

That's a matter of opinion. Personally, I think that *declarations* as
in `forward MyClass` are not especially Pythonic, although at least
your declaration also creates a real class object. But having two ways
to create a class (three if we count type()) is not especially Pythonic
either.


> As a side benefit, `forward class` and `continue class` syntax enables
> rudimentary separation of "interface" from "implementation", at least for
> classes.

I don't think so. The `forward class` syntax doesn't define any part of
the interface except the class' name. That's so rudimentary that it
doesn't deserve the name "interface".

"What's the Node class' API?"

"It's called Node."


> Syntax
> ------
>
> The `forward class` statement is the same as the `class` statement,
> except it doesn't end with a colon and is not followed by an indented block.

Or we could end it with a colon and follow it by an indented block, and
then dispose of the unnecessary "forward" keyword :-)



> Using this new syntax, the forward-reference problem illustrated
> in the *Rationale* section above is now easy to solve:

It's already easy to solve. Aside from the two PEPs, we have stringified
forward references and type comments:

class A:
value: "B" = None
value = None # type: B

If this is your most compelling example, I don't think this feature is
needed.



> `forward class X` declares a class, but the class is explicitly
> not yet fully defined.  It won't be fully defined and ready to
> be instantiated until after the corresponding `continue class`
> statement.  We'll refer to a class object in this state as
> a "forward-declared class object".  How does this object behave?
>
> As per the "consenting adults" rule, the forward-declared class
> object must permit most operations.  You should be able to examine
> the object, compare it to other objects, inspect some attributes
> (`__name__`, `__mro__`, `__class__`), and even set attributes.
>
> However, the user isn't permitted to instantiate a forward-declared
> class object until after the corresponding `continue class X`.

Why not?

Since you explicitly allow the user to instantiate the class by first
removing the `__forward__` dunder, this reminds me of the ancient Apple
Mac file system which had a copy-protect bit that even Apple called "the
Bozo bit". If the Bozo bit was set, you couldn't copy the file. But the
user could just unset the Bozo bit and then copy it.



> #### Semantics of `continue class`
>
> `continue class` may only be run on a class once.
> (As Eric V. Smith pointed out in response to an early version of
> this proposal, allowing multiple "continue" declarations on the
> same class would lead directly to language-condoned monkey-patching.)

We already have language-condoned monkey-patching.

If we can do C.attr = value, *and we can*, that's monkey-patching.



--
Steve
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/F2P2MUFKRCDPV2MLHGVFSU7N3A7ALDJZ/

Chris Angelico

unread,
Apr 23, 2022, 1:10:25 AM4/23/22
to pytho...@python.org
On Sat, 23 Apr 2022 at 12:50, Larry Hastings <la...@hastings.org> wrote:
>
>
> On 4/22/22 19:17, Chris Angelico wrote:
>
> I'm unsure about the forward class. How is it different from subclassing an ABC?
>
> They're just different objects. A subclass of an ABC is either itself another abstract base class, which will never be instantiatable, or a non-abstract class, which is immediately instantiatable. A forward-declared class object is not currently instantiatable, and is not fully defined, but will become fully defined and instantiatable after the matching "continue class" statement.
>

Ah okay, I think I had the idea that an ABC could demand certain
methods, but I think I'm mismatching semantics when it comes to
half-implementing something. Anyhow, it's a class, with some special
features (notably that you can't instantiate it).

>
> What happens if you try to continue a non-forward class?
>
> From the proto-PEP:
>
> Executing a `continue class` statement with a class defined by the `class` statement raises a `ValueError` exception.
>
> And also:
>
> It's expected that knowledgeable users will be able to trick Python into executing `continue class` on the same class multiple times by interfering with "dunder" attributes. The same tricks may also permit users to trick Python into executing `continue class` on a class defined by the `class` statement. This is undefined and unsupported behavior, but Python will not prevent it.
>

Huh. I skimmed back and forth looking, and did a search, and didn't
find that. My bad.

It seems odd that you define a blessed way of monkeypatching a class,
but then demand that it can only be done once unless you mess with
dunders. Why not just allow multiple continuations?

And if multiple continuations are permitted, there's really no
difference between a forward class and a regular class. It'd be a
"non-instantiable class", which is an aspect which can be removed by
reopening the class, but otherwise it's just like any other class.

ChrisA
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/AELFL53RTT2GXOZ6NA7BFEJUE2BH67UR/

Larry Hastings

unread,
Apr 23, 2022, 1:10:33 AM4/23/22
to pytho...@python.org


On 4/22/22 20:58, Steven D'Aprano wrote:
On Fri, Apr 22, 2022 at 06:13:57PM -0700, Larry Hastings wrote:

This PEP proposes an additional syntax for declaring a class which splits
this work across two statements:
* The first statement is `forward class`, which declares the class and binds
  the class object.
* The second statement is `continue class`, which defines the contents
  of the class in the "class body".

To be clear: `forward class` creates the official, actual class object.
Code that wants to take a reference to the class object may take references
to the `forward class` declared class, and interact with it as normal.
However, a class created by `forward class` can't be *instantiated*
until after the matching `continue class` statement finishes.
Since the "forward class" is a real class, 

It's a "forward-declared class object".  It's the real class object, but it hasn't been fully initialized yet, and won't be until the "continue class" statement.


it doesn't need any new 
syntax to create it. Just use plain old regular class syntax.

    class X(object):
        """Doc string"""
        attribute = 42

And now we have our X class, ready to use in annotations.

To add to it, or "continue class" in your terms, we can already do this:

    X.value = "Hello World"

But if X has a metaclass that defines __new__ , "value" won't be defined yet, so metaclass.__new__ won't be able to react to and possibly modify it.  Similarly for metaclass.__init__ and BaseClass.__init_subclass__.

So, while your suggested technique doesn't "break" class creation per se, it prevents the user from benefiting from metaclasses and base classes using these advanced techniques.


Counter proposal:

`continue class expression:` evaluates expression to an existing class 
object (or raises an exception) and introduces a block. The block is 
executed inside that class' namespace, as the `class` keyword does, 
except the class already exists.

If "continue class" is run on an already-created class, this breaks the functionality of __prepare__, which creates the namespace used during class body execution and is thrown away afterwards.  The "dict-like object" returned by __prepare__ will have been thrown away by the time "continue class" is executed.

Also, again, this means that the contents added to the class in the "continue class" block won't be visible to metaclass.__new__, metaclass.__init__, and BaseClass.__init_subclass__.

Also, we would want some way of preventing the user from running "continue class" multiple times on the same class--else we accidentally condone monkey-patching in Python, which we don't want to do.


Isn't this case solved by either forward references:

    class A:
        value: "B"

or by either of PEP 563 or PEP 649?

It is, but:

a) manual stringizing was rejected by the community as too tiresome and too error-prone (the syntax of the string isn't checked until you run your static type analysis tool).  Also, if you need the actual Python value at runtime, you need to eval() it, which causes a lot of headaches.

b) PEP 649 doesn't solve this only-slightly-more-advanced case:

@dataclass
class A:
  value: B

@dataclass
class B:
  value: A

as the dataclass decorator examines the contents of the class, including its annotations.

c) PEP 563 has the same "what if you need the actual Python value at runtime" problem as manual stringization, which I believe is why the SC has delayed its becoming default behavior.

Perhaps my example for b) would be a better example for the PEP.


That could become:

    class A:
        pass

    class B:
        value: A  # This is fine, A exists.

    A.value: B  # And here B exists, so this is fine too.

No new syntax is needed. This is already legal.

It's legal, but it doesn't set the annotation of "value" on A.  Perhaps this is just a bug and could be fixed.  (TBH I'm not sure what the intended semantics of this statement are, or where that annotation ends up currently.  I couldn't find it in A.__annotations__ or the module's __annotations__.  Is it just forgotten?)

I assert this approach will be undesirable to Python programmers.  This makes for two very-different feeling approaches to defining the members of a class.  One of the goals of my PEP was to preserve the existing "feel" of Python as much as possible.

Also, as previously mentioned, your technique prevents "A.value", and all other attributes and methods set using this technique, from being visible to metaclass.__new__, metaclass.__init__, and BaseClass.__init_subclass__.


This proposed  `forward class` / `continue class` syntax should permit
solving *every* forward-reference and circular-reference problem faced
in Python,
I think that's overselling the concept.

Okay, perhaps I should have said "the forward-reference and circular-reference problems of class definitions" or something like that.  I'll adjust the text for the second draft.


using an elegant and Pythonic new syntax.
That's a matter of opinion.

Yes.  Are PEPs not permitted to express opinions?


As a side benefit, `forward class` and `continue class` syntax enables
rudimentary separation of "interface" from "implementation", at least for
classes.
I don't think so. The `forward class` syntax doesn't define any part of 
the interface except the class' name.

It also defines the base classes and metaclass, as well as some other simple metadata ("__file__"), all which may be of interest to external consumers of the object.

If you don't like the PEP calling this 'the rudimentary separation of 'interface' from 'implementation'", what is your counter-proposal?


However, the user isn't permitted to instantiate a forward-declared
class object until after the corresponding `continue class X`.
Why not?

Because the class object is not fully initialized yet; it's a "forward-declared class object".  I thought my PEP was pretty clear on that point.

Feel free to make your counter-proposal, but it's incoherent to debate the statements of my proposal as if they're statements about your counter-proposal.


Since you explicitly allow the user to instantiate the class by first 
removing the `__forward__` dunder,

"allowing" and "condoning" are two different things.  Python allows a lot of things that are not condoned.


/arry

Larry Hastings

unread,
Apr 23, 2022, 1:29:09 AM4/23/22
to pytho...@python.org


On 4/22/22 22:03, Chris Angelico wrote:
Anyhow, [a forward-defined class object is] a class, with some special
features (notably that you can't instantiate it).

Yes.  Specifically, here's my intention for "forward-defined class objects": you can examine some generic dunder values (__name__, __mro__), and you can take references to it.  You can't instantiate it or meaningfully examine its contents, because it hasn't been fully initialized yet.


It seems odd that you define a blessed way of monkeypatching a class,
but then demand that it can only be done once unless you mess with
dunders. Why not just allow multiple continuations?

I think monkeypatching is bad, and I'm trying to avoid Python condoning it.

On that note, the intent of my proposal is that "continue class" is not viewed as "monkeypatching" the class, it's the second step in defining the class.

I considered attempting to prevent the user modifying the "forward-declared class object".  But a) that just seemed like an arms race with the user--"oh yeah? well watch THIS!" and b) I thought the Consenting Adults rule applied.

Still, it's not the intent of my PEP to condone or facilitate monkeypatching.


/arry

Chris Angelico

unread,
Apr 23, 2022, 1:48:51 AM4/23/22
to pytho...@python.org
I guess perception is everything. This really *is* monkeypatching; you
have an object, and that object will be mutated, and those mutations
are defined by adding functionality to the class. It's not
fundamentally different from X.__dict__.update() with a pile of new
methods etc. The only difference is that you call it something
different.

Is that sufficient justification? I'm unsure.

ChrisA
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/DIEGQM5ARFGIYPWG34CQODAENKNBVUET/

Mehdi2277

unread,
Apr 23, 2022, 2:47:59 AM4/23/22
to pytho...@python.org
My main question for this approach is how would this work with type checkers? Is there any restriction that forward class's continuation must appear in same module? If it's allowed that a forward class may be continued in a different module I do not see how type checker like mypy/pyright could handle that. Classes are generally viewed as closed and fully defined within type checker. Monkey patching at runtime later is not supported. Libraries/techniques that rely on extending a class later tend to be sources of type errors/poor type checking coverage. ABC.register for dynamically declaring subclass relationship is similarly not supported by checkers. So without constraint that forward class must be continued within same module I think it's unlikely any type checker would support this technique. As main motivation is related to type annotations I think usage should be supported by type checkers.

One other edge case here is how would you forward declare an annotation for a type that like this,

if TYPE_CHECKING:
import numpy

def f(x: numpy.ndarray) -> None:
...

forward declaring ndarray here would not make numpy.ndarray available. Would you forward declare modules? Is that allowed? Looking at sourcegraph I do see a some module imports in TYPE_CHECKING blocks for purpose of using them as annotations.

I'm confused in general how if TYPE_CHECKING issue is handled by this approach. Usually class being imported in those blocks is defined normally (without continue class) somewhere else.

My current leaning lies towards pep 649 + smart eval strategy by get_type_hints that Carl proposed.
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/XNYKAEI6TSZU2EKQEM6AXDZOMVD3CPLS/

Steven D'Aprano

unread,
Apr 23, 2022, 3:25:51 AM4/23/22
to pytho...@python.org
On Sat, Apr 23, 2022 at 06:41:23AM -0000, Mehdi2277 wrote:

> My main question for this approach is how would this work with type
> checkers? Is there any restriction that forward class's continuation
> must appear in same module? If it's allowed that a forward class may
> be continued in a different module I do not see how type checker like
> mypy/pyright could handle that.

Larry said that the name that follows `continue class` is an expression,
so that something like this is allowed:

import mymodule
continue class mymodule.X:
def method(self):
pass

so yes, you can continue classes in other modules. He said that could be
used as a very primitive form of separation of interface and
implementation, by putting the `forward class` in one module and the
`continue` in another.


--
Steve
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/N3YIF2FJARSE73M6VJX6UNU776FS6QC4/

Larry Hastings

unread,
Apr 23, 2022, 3:53:38 AM4/23/22
to pytho...@python.org


On 4/22/22 23:41, Mehdi2277 wrote:
My main question for this approach is how would this work with type checkers?

It would be new syntax for Python, so type checkers would have to understand it.


Is there any restriction that forward class's continuation must appear in same module?

No.


If it's allowed that a forward class may be continued in a different module I do not see how type checker like mypy/pyright could handle that. Classes are generally viewed as closed and fully defined within type checker. Monkey patching at runtime later is not supported.

If it became official Python syntax, I suspect they'd figure out a way to support it.

They might require that the expression used in the "continue class" statement map to the original "forward class" declaration, e.g. they might stipulate that they don't support this:

forward class X
random_name = X

continue class random_name:
    ...

But rather than speculate further, perhaps someone who works on one of the static type analysis checkers will join the discussion and render an informed opinion about how easy or hard it would be to support "forward class" and "continue class".


One other edge case here is how would you forward declare an annotation for a type that like this,

if TYPE_CHECKING:
  import numpy

def f(x: numpy.ndarray) -> None:
 ...

forward declaring ndarray here would not make numpy.ndarray available.

In this case, adding forward declarations for classes in "numpy" would be up to the "numpy" module.  One approach might look more like this:

import numpy # contains forward declarations
if TYPE_CHECKING:
    import numpy.impl

Though numpy presumably couldn't do this while they still supported older versions of Python.  That's one downside of using new syntax--you can't use it until you stop support for old versions of Python that predate it.


Would you forward declare modules? Is that allowed?

I haven't proposed any syntax for forward-declaring modules, only classes.


I'm confused in general how if TYPE_CHECKING issue is handled by this approach. Usually class being imported in those blocks is defined normally (without continue class) somewhere else.

My proposal should mate well with "if TYPE_CHECKING".  You would define your forward classes in a module that does get imported, but leave the continue classes in a separate module that is only imported "if TYPE_CHECKING", as per my example with "numpy" above.


/arry

Steven D'Aprano

unread,
Apr 23, 2022, 3:58:46 AM4/23/22
to pytho...@python.org
On Fri, Apr 22, 2022 at 10:09:33PM -0700, Larry Hastings wrote:

[Larry]
> >>To be clear: `forward class` creates the official, actual class object.
> >>Code that wants to take a reference to the class object may take
> >>references
> >>to the `forward class` declared class, and interact with it as normal.
> >>However, a class created by `forward class` can't be *instantiated*
> >>until after the matching `continue class` statement finishes.

[Steve (me)]
> >Since the "forward class" is a real class,
>
> It's a "forward-declared class object".  It's the real class object, but
> it hasn't been fully initialized yet, and won't be until the "continue
> class" statement.

The only thing that makes it not fully initialised is that it has a bozo
bit dunder "__forward__" instructing the interpreter to disallow
instantiation. Yes?

If I take that class object created by `forward class X`, and delete the
dunder, there is no difference between it and other regular classes. Am
I correct?

So your reasoning is circular: you give it a dunder marking it as a
"forward-declared class object" to prevent it from being instantiated,
but the only reason it can't be instantiated is that it has the dunder.

I won't respond to the rest of your post until you have clarified the
above, in case I have misunderstood.


--
Steve
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/B7BZDPTWQTZ67PO7AB2BPOBJRDKAFCDZ/

Steven D'Aprano

unread,
Apr 23, 2022, 4:17:50 AM4/23/22
to pytho...@python.org
On Sat, Apr 23, 2022 at 12:46:37AM -0700, Larry Hastings wrote:

> But rather than speculate further, perhaps someone who works on one of
> the static type analysis checkers will join the discussion and render an
> informed opinion about how easy or hard it would be to support "forward
> class" and "continue class".

No offense Larry, but since this proto-PEP is designed to help the
typing community (I guess...) shouldn't you have done that before
approaching Python-Dev with the proposal?

I might have missed it, but I don't think I've seen this discussed on
the typing-sig mailing list.

Or have I misunderstood? If typing is not driving this, what is?


> >One other edge case here is how would you forward declare an annotation
> >for a type that like this,
> >
> >if TYPE_CHECKING:
> > import numpy
> >
> >def f(x: numpy.ndarray) -> None:
> > ...
> >
> >forward declaring ndarray here would not make numpy.ndarray available.
>
> In this case, adding forward declarations for classes in "numpy" would
> be up to the "numpy" module.  One approach might look more like this:
>
> import numpy # contains forward declarations
> if TYPE_CHECKING:
>     import numpy.impl

Wouldn't that be a massively breaking change? Anyone who does:

from numpy import ndarray

will get the forward-declared class object instead of the fully
initialised class object, leading to all sorts of action-at-a-distance
bugs.


from numpy import ndarry
a = ndarray([1, 2]) # Fails, because its a FDCO

import some_random_module # which happens to import numpy.impl
# which finishes the initialisation of the class object

a = ndarray([1, 2]) # and this now magically works!


--
Steve
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/5NPG3AG2GYDY7S3RBZD3XTUQDD7WQEQU/

Larry Hastings

unread,
Apr 23, 2022, 4:41:17 AM4/23/22
to pytho...@python.org


On 4/23/22 00:53, Steven D'Aprano wrote:
It's a "forward-declared class object".  It's the real class object, but 
it hasn't been fully initialized yet, and won't be until the "continue 
class" statement.
The only thing that makes it not fully initialised is that it has a bozo 
bit dunder "__forward__" instructing the interpreter to disallow 
instantiation. Yes?

If I take that class object created by `forward class X`, and delete the 
dunder, there is no difference between it and other regular classes. Am 
I correct?

No, there are several differences.

  • It still has the "dict-like object" returned by metaclass.__prepare__, rather than its final dict.
  • Its class body hasn't been called yet, so it likely doesn't have any of its important dunder methods.
  • It hasn't had its  BaseClass.__init_subclass__ called yet.
  • It hasn't had its metaclass.__init__ called yet.

The "forward-declared class object" is in a not-yet-fully initialized state, and is not ready for use as a class.

From my perspective, the "__forward__" attribute is an internal implementation detail, and something that user code should strictly leave alone.  But if it's considered too dangerous to expose to users, we could hide it in the class object and not expose it to users.  I'm not convinced that's the right call; I think the Consenting Adults rule still applies.  Python lets you do crazy things like assigning to __class__, and resurrecting objects from inside their __del__; manually removing __forward__ seems like it falls into the same category.  It's not recommended, and we might go so far as to say doing that results in undefined behavior.  But Python shouldn't stand in your way if you really think you need to do it for some reason.


/arry

Larry Hastings

unread,
Apr 23, 2022, 4:54:46 AM4/23/22
to pytho...@python.org


On 4/23/22 01:14, Steven D'Aprano wrote:
On Sat, Apr 23, 2022 at 12:46:37AM -0700, Larry Hastings wrote:

But rather than speculate further, perhaps someone who works on one of 
the static type analysis checkers will join the discussion and render an 
informed opinion about how easy or hard it would be to support "forward 
class" and "continue class".
No offense Larry, but since this proto-PEP is designed to help the 
typing community (I guess...) shouldn't you have done that before 
approaching Python-Dev with the proposal?

The perfect is the enemy of the good.  Like I said, I wanted to get this out there before the Language Summit, and I just ran out of time.  I think there's also some sort of typing summit next week?  I'm not really plugged in to the static type analysis world--I don't use it in any of my projects.


Wouldn't that be a massively breaking change? Anyone who does:

    from numpy import ndarray

will get the forward-declared class object instead of the fully 
initialised class object, leading to all sorts of action-at-a-distance 
bugs.

I wasn't recommending The Famous numpy Project do this exact thing, it was an abstract example using the name "numpy".  I didn't think this was a real example anyway, as I was assuming that most people who import numpy don't do so in an "if TYPE_CHECKING:" block.

Separating the forward class declaration from the continue class implementation in the actual "numpy" module itself is probably not in the cards for a while, if ever.  But perhaps numpy could do this:

import numpy.forward
if TYPE_CHECKING:
    import numpy

In this case, the "numpy" module would also internally "import numpy.forward", and would contain the "continue class" statements for the forward-declared classes in "numpy.forward".

There are lots of ways to solve problems with the flexibility afforded by the proposed "forward class" / "continue class" syntax.  Perhaps in the future you'll suggest some of them!


/arry

Terry Reedy

unread,
Apr 23, 2022, 6:13:09 AM4/23/22
to pytho...@python.org
On 4/22/2022 11:16 PM, Larry Hastings wrote:

> So I still prefer "forward class".
>
> I don't think it's as clear as "forward class"

'forward class' for an incomplete class is not at all clear to me. It
is not clear to me which part of speech you intend it to be: noun, verb,
adjective, or adverb. You must have some experience with 'forward' in a
different context that makes it clearer to you.

--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/4BB7UPSE26RFYUMX4PZOJHHQYKCYAYAR/

Larry Hastings

unread,
Apr 23, 2022, 6:27:35 AM4/23/22
to pytho...@python.org


On 4/23/22 03:10, Terry Reedy wrote:
On 4/22/2022 11:16 PM, Larry Hastings wrote:

So I still prefer "forward class".

I don't think it's as clear as "forward class"

'forward class' for an incomplete class is not at all clear to me.  It is not clear to me which part of speech you intend it to be: noun, verb, adjective, or adverb.  You must have some experience with 'forward' in a different context that makes it clearer to you.


It's a reference to the term "forward declaration":

https://en.wikipedia.org/wiki/Forward_declaration


/arry

Larry Hastings

unread,
Apr 23, 2022, 7:08:05 AM4/23/22
to Python-Dev
I should have said "numpy_forward", not "numpy.forward".  I changed my mind at the last second as I was writing that email, and momentarily forgot that when you import x.y you implicitly import x.


/arry

Jelle Zijlstra

unread,
Apr 23, 2022, 10:01:36 AM4/23/22
to Larry Hastings, Python-Dev
I don't think this proposal is a good solution for the problems of static typing users.

El vie, 22 abr 2022 a las 18:16, Larry Hastings (<la...@hastings.org>) escribió:
But the explosion of static type analysis in Python, particularly with
the `typing` module and the `mypy` tool, has made circular definition-time
dependencies between classes commonplace--and much harder to solve.  Here's
one simple example:

```Python
     class A:
         value: B

     class B:
         value: A
```

So to reiterate, your proposal would be to write this as:

forward class B:
    pass

class A:
    value: B

continue class B:
    value: A

While the current workaround is:

class A:
    value: "B"

class B:
     value: "A"

I don't think I would write the "forward class" version if I had the choice. It's clunkier and requires more refactoring if I change my mind about whether the `value` attribute should exist.

I'd prefer if we found some elegant way to get the natural way to write this code to work, which is the way you wrote it in your post (with no explicit declarations or stringifications). Carl Meyer's idea at https://github.com/larryhastings/co_annotations/issues/2#issuecomment-1092432875 is a very promising approach that should allow us to do that.

I also agree with Mehdi2277's concern that this feature would be difficult for static type checkers to fully implement, because class declaration and implementation may be widely separated. To be sure, static checkers can put in additional restrictions, like requiring declaration and implementation to be in the same module, but it would be unfortunate if a feature designed to help users of static typing actually makes it harder to adopt it.

-----

That said, one nice thing about the proposal is that it can help with forward references outside of annotations, a problem which neither PEP 563 nor 649 currently solves. Those include type aliases (MyAlias = Sequence["A | B"]) and TypeVar bounds (Typevar("AT", bound="A | B")). However, it doesn't solve the problem for base classes. For example, str is conceptually defined as `class str(Sequence["str"]):`. A forward reference can't make `str` defined when the bases are evaluated, because bases are resolved at the `forward class` stage.
 

Tobias HT

unread,
Apr 23, 2022, 11:34:00 AM4/23/22
to pytho...@python.org
This seems like it would solve a huge problem that I've often faced myself, but by introducing something that might later on cause problems I anticipate. 

Like some people suggested earlier, introducing new keyword in python is not a good solution, I feel it just adds to the already existing bulk and increases complexity.

Also Like some else mentioned again, this solution doesn't seem pythonic. I don't know what they meant by that, but in hindsight, it does really not feel pythonic.

On the other hand, there's something I've been seeing around. I don't know if it was introduced by Mypy or something, but its the use of declaration files. I think they are saved as pyi files. They just have the declaration of a python object, be it class or variable. What if we just found a way of reusing that instead?

Eric V. Smith

unread,
Apr 23, 2022, 12:56:36 PM4/23/22
to Jelle Zijlstra, Larry Hastings, Python-Dev
On 4/23/2022 9:55 AM, Jelle Zijlstra wrote:

However, it doesn't solve the problem for base classes. For example, str is conceptually defined as `class str(Sequence["str"]):`. A forward reference can't make `str` defined when the bases are evaluated, because bases are resolved at the `forward class` stage.
 

Larry's second email "Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object" discusses a possibility to move the bases and metaclasses to the "continue class" stage. It also has the advantage of not changing the behavior of __new__, and I think is in general easier to reason about. He and I have discussed this approach, but neither of have looked at in enough detail to know if the implementation is possible. Some of the concerns are noted in that email.

Eric

Eric V. Smith

unread,
Apr 23, 2022, 12:57:52 PM4/23/22
to pytho...@python.org
On 4/23/2022 3:28 AM, Tobias HT wrote:


On the other hand, there's something I've been seeing around. I don't know if it was introduced by Mypy or something, but its the use of declaration files. I think they are saved as pyi files. They just have the declaration of a python object, be it class or variable. What if we just found a way of reusing that instead?

As they currently exist, stub files (.pyi files) don't contain enough information. In particular, they don't have the metaclass information. This could be changed, but at that point you basically have the "forward" declaration, but it's hidden away where the interpreter can't see it.

Eric

Larry Hastings

unread,
Apr 23, 2022, 1:31:19 PM4/23/22
to Jelle Zijlstra, Python-Dev


On 4/23/22 06:55, Jelle Zijlstra wrote:
So to reiterate, your proposal would be to write this as:

forward class B:
    pass

class A:
    value: B

continue class B:
    value: A

Not quite; the "forward class" statement doesn't have a colon or a class body.  This would be written as:

forward class B



class A:
    value: B

continue class B:
    value: A


While the current workaround is:

class A:
    value: "B"

class B:
     value: "A"

In this example, with two toy classes in one file, it shouldn't be necessary to quote the annotation in B.  So all you need is the quotes around the first annotation:

class A:
    value: "B"

class B:
     value: A


I don't think I would write the "forward class" version if I had the choice. It's clunkier and requires more refactoring if I change my mind about whether the `value` attribute should exist.

In this toy example, it adds an extra line.  Describing that as "clunky" is a matter of opinion; I disagree and think it's fine.

But the real difference is when it comes to larger codebases.  If classes "A" and "B" are referenced dozens or even hundreds of times, you'd have to add quote marks around every annotation that references one (both?).  Manual stringization of large codebases was sufficiently disliked as to have brought about the creation and acceptance of PEP 563.  Judicious use of the "forward class" statement should obviate most (all?) the manual stringizing in these codebases.


/arry

Larry Hastings

unread,
Apr 23, 2022, 1:55:22 PM4/23/22
to Eric V. Smith, Jelle Zijlstra, Python-Dev


On 4/23/22 08:57, Eric V. Smith wrote:
On 4/23/2022 9:55 AM, Jelle Zijlstra wrote:

However, it doesn't solve the problem for base classes. For example, str is conceptually defined as `class str(Sequence["str"]):`. A forward reference can't make `str` defined when the bases are evaluated, because bases are resolved at the `forward class` stage.
 

Larry's second email "Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object" discusses a possibility to move the bases and metaclasses to the "continue class" stage. It also has the advantage of not changing the behavior of __new__, and I think is in general easier to reason about.


Let me expound on Eric's statement for a bit.  Moving the base classes and metaclass to the "continue" statement might permit the self-referential "str" definition suggested above:

forward class str

...

continue class str(Sequence[str]):
    ...

Though I suspect this isn't viable, or at least not today.  I'm willing to bet a self-referential definition like this would result in an infinite loop when calculating the MRO.


I don't have a strong sense of whether it'd be better to have the base classes and metaclass defined with the "forward" declaration or the "continue" declaration.  The analogous syntax in C++ has the base classes defined with the "continue" class.

Actually, that reminds me of something I should have mentioned in the proto-PEP.  If the "class proxy" version is viable and desirable, and we considered moving the base classes and metaclass down to the "continue" statement, that theoretically means we could drop the "forward" and "continue" keywords entirely.  I prefer them, simply because it makes it so explicit what you're reading.  But the equivalent syntax in C++ doesn't bother with extra keywords for either the "forward" declaration or the "continue" declaration, and people seem to like it fine.  Using that variant of the syntax, the toy example from the PEP would read as follows:

class A

class B:
   value: A

class A:
   value: B

If you're speed-reading here and wondering "wait, how is this not ambiguous?", note that in the first line of the example, the "class" statement has no colon.  Also, it would never have parentheses or decorators.  The forward declaration of a class would always be just "class", followed by a name, followed by a newline (or comment).


/arry

Rob Cliffe via Python-Dev

unread,
Apr 23, 2022, 5:40:01 PM4/23/22
to pytho...@python.org
UGH!

I thought there was a general understanding that when typing was added
to Python, there would be no impact, or at least minimal impact, on
people who didn't use it.  (Raises hand.)
Now we see an(other) instance of intention creep.
Rob Cliffe

On 23/04/2022 02:13, Larry Hastings wrote:
>
>
> This document is a loose proto-PEP for a new "forward class" /
> "continue class" syntax.  Keep in mind, the formatting is a mess. If I
> wind up submitting it as a real PEP I'll be sure to clean it up first.
>
>
> /arry
>
> --------------------------------------
>
>
> PEP XXXX: Forward declaration of classes
>
> Overview
> --------
>
> Python currently has one statement to define a class, the `class`
> statement:
>
> ```Python
>     class X():
>         # class body goes here
>         def __init__(self, key):
>             self.key = key
> ```
>
> This single statement declares the class, including its bases and
> metaclass,
> and also defines the contents of the class in the "class body".
>
> This PEP proposes an additional syntax for declaring a class which splits
> this work across two statements:
> * The first statement is `forward class`, which declares the class and
> binds
>   the class object.
> * The second statement is `continue class`, which defines the contents
>   of the class in the "class body".
>
> To be clear: `forward class` creates the official, actual class object.
> Code that wants to take a reference to the class object may take
> references
> to the `forward class` declared class, and interact with it as normal.
> However, a class created by `forward class` can't be *instantiated*
> until after the matching `continue class` statement finishes.
>
> Defining class `X` from the previous example using this new syntax
> would read
> as follows:
>
> ```
>     forward class X()
>
>     continue class X:
>         # class body goes here
>         def __init__(self, key):
>             self.key = key
> ```
>
> This PEP does not propose altering or removing the traditional `class`
> statement;
> it would continue to work as before.
>
>
> Rationale
> ---------
>
> Python programmers have had a minor problem with classes for years:
> there's
> no way to have early-bound circular dependencies between objects. If A
> depends on B, and B depends on A, there's no linear order that allows
> you to cleanly declare both.
>
> Most of the time, the dependencies were in late-binding code, e.g. A
> refers
> to B inside a method.  So this was rarely an actual problem at
> runtime.  When
> this problem did arise, in code run at definition-time, it was usually
> only
> a minor headache and could be easily worked around.
>
> But the explosion of static type analysis in Python, particularly with
> the `typing` module and the `mypy` tool, has made circular
> definition-time
> dependencies between classes commonplace--and much harder to solve. 
> Here's
> one simple example:
>
> ```Python
>     class A:
>         value: B
>
>     class B:
>         value: A
> ```
>
> An attribute of `B` is defined using a type annotation of `A`, and an
> attribute of `A` is defined using a type annotation of `B`. There's
> no order to these two definitions that works; either `A` isn't defined
> yet, or `B` isn't defined yet.
>
> Various workarounds and solutions have been proposed to solve this
> problem,
> including two PEPs: PEP 563 (automatic stringized annotations) and PEP
> 649
> (delayed evaluation of annotations using functions).
> But nothing so far has been both satisfying and complete; either it
> is wordy and clumsy to use (manually stringizing annotations), or it
> added restrictions and caused massive code breakage for runtime use of
> annotations (PEP 563), or simply didn't solve every problem (PEP 649).
> This proposed  `forward class` / `continue class` syntax should permit
> solving *every* forward-reference and circular-reference problem faced
> in Python, using an elegant and Pythonic new syntax.
>
> As a side benefit, `forward class` and `continue class` syntax enables
> rudimentary separation of "interface" from "implementation", at least for
> classes.  A user seeking to "hide" the implementation details of their
> code could put their class definitions in one module, and the
> implementations of those classes in a different module.
>
> This new syntax is not intended to replace the traditional `class`
> declaration syntax in Python.  If this PEP were accepted, the `class`
> statement would still be the preferred mechanism for creating classes
> in Python; `forward class` should only be used when it confers some
> specific benefit.
>
>
> Syntax
> ------
>
> The `forward class` statement is the same as the `class` statement,
> except it doesn't end with a colon and is not followed by an indented
> block.
> Without any base classes or metaclass, the `forward class` statement is
> as follows:
>
> ```
>     forward class X
> ```
>
> This would declare class `X`.
>
> If `X` needs base classes or metaclass, the corresponding `forward
> class` statement
> would be as follows, rendered in a sort of "function prototype" manner:
>
> ```
>     forward class X(*bases, metaclass=object, **kwargs)
> ```
>
> The `continue class` statement is similar to a `class` statement
> without any bases or metaclass.  It ends with a colon,
> and is followed by the "class body":
>
>     continue class X:
>         # class body goes here
>         pass
>
> One important difference: the `X` in `continue class X:` is not a *name*,
> it's an *expression*.  This code is valid:
>
> ```
>     forward class X()
>     snodgrass = X
>
>     continue class snodgrass:
>         # class body goes here
>         pass
> ```
>
> as well as this:
>
> ```
>     import my_module
>
>     continue class my_module.X:
>         # class body goes here
>         pass
> ```
>
> Using this new syntax, the forward-reference problem illustrated
> in the *Rationale* section above is now easy to solve:
>
> ```Python
>     forward class B
>
>     class A:
>         value: B = None
>
>     continue class B:
>         value: A = None
> ```
>
> One final note.  Why must the base and metaclass be declared
> with the `forward class` statement?  The point of this new
> syntax is to allow creating the real class object, permitting
> users of the class to take references to it early, before it's
> fully defined.  And the class could be declared with a
> metaclass, and the metaclass could have a `__new__`, which
> means it's responsible for creating the class object, and
> this syntax would have to preserve that behavior.
>
> (This small detail is about to complicate this proposal a great
> deal!)
>
>
> #### Semantics of forward-declared class objects
>
> `forward class X` declares a class, but the class is explicitly
> not yet fully defined.  It won't be fully defined and ready to
> be instantiated until after the corresponding `continue class`
> statement.  We'll refer to a class object in this state as
> a "forward-declared class object".  How does this object behave?
>
> As per the "consenting adults" rule, the forward-declared class
> object must permit most operations.  You should be able to examine
> the object, compare it to other objects, inspect some attributes
> (`__name__`, `__mro__`, `__class__`), and even set attributes.
>
> However, the user isn't permitted to instantiate a forward-declared
> class object until after the corresponding `continue class X`.
> We ensure this with a new dunder attribute, `__forward__`,
> which if present tells the Python runtime that this is a
> forward-declared class object.  The `continue class` statement
> would delete this attribute from the object, after which it
> could be instantiated.
>
> (Users could work around this constraint, or even delete `__forward__`
> if they so chose--again, the "consenting adults" rule applies.)
>
> It's explicitly permissible to create a forward-declared class
> object that never gets finished with a `continue class` statement.
> If all you need is an object that represents the class--say,
> to satisfy static type annotation use cases--a forward-declared
> class object works fine.
>
> A subsequent section will address the complexities of
> how `forward class` and `continue class` interact with metaclasses.
> For now, a note about forward-declared class objects declared with
> a metaclass implementing `__prepare__`.  The forward-declared class
> object *dict* will be the "dict-like object" returned by the
> `metaclass.__prepare__()` method.  This "dict-like object" won't
> be processed and discarded until after `continue class` processes
> the class body and calls the appropriate methods in the metaclass.
>
>
> #### Semantics of `continue class`
>
> `continue class` may only be run on a class once.
> (As Eric V. Smith pointed out in response to an early version of
> this proposal, allowing multiple "continue" declarations on the
> same class would lead directly to language-condoned monkey-patching.)
>
>
> #### Decorators
>
> Both the `forward class` and `continue class` statements
> support decorators, and the user may use decorators with either
> or both statements for the same class.  But now that we've
> split the responsibilities of the `class` statement between
> these two new statements, which decorator goes with which
> statement becomes a novel concern.  In general, decorators
> that don't examine the contents of the class, but simply
> want to register the class object and its name, can decorate
> the `forward class` statement.  Also, class decorators that
> want to return a different object for the class should decorate
> `forward class`.  But all decorators that meaningfully examine
> the contents of the class should decorate the `continue class`
> statement.
>
> Unfortunately, there are some decorators that can't work properly
> with either `forward class` *or* `continue class`: a decorator
> that meaningfully examine the declared contents of that class, but
> also return an object other than the original class passed in.
> In that case, the user cannot declare this class with
> `forward class`; they must declare it with the conventional `class`
> statement.
>
> #### __slots__
>
> This leads us to an example of a decorator that, as of 3.10,
> wouldn't be usable with classes declared by `forward class`.
> It's the new 3.10 feature `@dataclass(slots=True)`.  When called
> in this way, dataclass examines the attributes of the class it
> has decorated, dynamically constructs a new class using `__slots__`,
> and returns this new class.
>
> Since this decorator meaningfully examines the class, it must
> be used with `continue class`.  But, this decorator also returns
> an object other the original class, which means it's inapproriate
> for `continue class` and should be called with `forward class`.
> What to do?
>
> We have a separate idea to ameliorate this specific situation.
> Right now, a class that uses `__slots__` *must* define them in
> the class body, as that member is read before the class name is
> bound (or before any descriptors are run).  But we can simply
> relax that, and make processing `__slots__` lazy, so that it
> isn't examined until the first time the class is *instantiated.*
> This would mean `@dataclass(slots=True)` could simply return the
> original class, and thus would work fine when decorating a
> `continue class` statement.
>
>
> #### Metaclasses
>
> The existing semantics of metaclasses present a thorny problem
> for `forward class` and `continue class`.  First, a review of
> how class definition works.
>
> Most of the mechanisms involved with defining a class happen
> internally to the interpreter.  However, there are a number
> of special methods (aka "dunder methods") called during class
> construction that are overridable by user code.  Empirical testing
> with the current version of Python (as of this writing, 3.10.4)
> reveals the order in which all this work is done.
>
> When Python executes the definition of this class:
>
> ```Python
>     class Foo(BaseClass, metaclass=MetaClass):
>         # class body is here
>         pass
> ```
>
> these events are visible in this order:
>
> 1. Python calls `MetaClass.__prepare__`.
> 2. Python executes the "class body" for class Foo.
> 3. Python calls `MetaClass.__new__`.
> 4. Python calls `BaseClass.__init_subclass__`.
>    (If multiple base classes define `__init_subclass__`,
>    they're called in MRO order.)
> 5. Python calls `MetaClass.__init__`.
> 6. The `class` statement binds the class object to the name `Foo`.
>
> The big problem this presents for `forward class`: the
> "class body" is executed before the `MetaClass.__new__`.
> This is necessary because one of the parameters to `MetaClass.__new__`
> is `namespace`, the "dict-like object" returned by
> `MetaClass.__prepare__`
> and initialized by executing the class body using that object
> as a sort of locals dictionary.
>
> This creates a chicken-and-egg problem: `forward class` needs
> to define the class object, but the class object is defined
> by `MetaClass.__new__`, and `MetaClass.__new__` can't run until
> after the class body, which we don't run until the `continue class`
> statement, which must be after `forward class`.
>
> The unfortunate but necessary solution: split `__new__` into
> two new special methods on metaclasses, `__new_forward__`
> and `__new_continue__`.  As a reminder, here's the prototype
> for `__new__` on a metaclass:
>
> ```Python
>     def __new__(metaclass, name, bases, namespace, **kwargs):
> ```
>
> The two new special methods would have the following prototypes:
>
> ```Python
>     def __new_forward__(metaclass, name, bases, namespace, **kwargs):
>
>     def __new_continue__(metaclass, cls, **kwargs):
> ```
>
> `__new_forward__` creates the class object.  It sets the `namespace`
> member as the class dict, but in general should not examine it
> contents.  (Specifically, `__new_forward__` cannot make any assumptions
> about whether or not the class body has been executed yet; more on this
> in a moment.)
>
> `__new_continue__` is guaranteed to be called after the
> class body has been executed.
>
> The externally-visible parts of class construction would
> run in a different order for classes constructed using
> `forward class` and `continue class`.  First, the visible
> interactions from the `forward class` statement:
>
> 1. Python calls `MetaClass.__prepare__`.
> 2. Python calls `MetaClass.__new_forward__`.
> 3. The `forward class` statement binds the (forward-declared)
>    class object to the name `Foo`.
>
> And here are the visible interactions from the
> `continue class` statement:
>
> 1. Python executes the class body.
> 2. Python calls `MetaClass.__new_continue__`.
> 3. Python calls `BaseClass.__init_subclass__`.
>    (If multiple base classes define `__init_subclass__`,
>    they're called in MRO order.)
> 4. Python calls `MetaClass.__init__`.
>
> It's important to note that, while `namespace` is passed
> in to `__new_forward__`, it's not yet initialized with the
> class body.  It's passed in here because the "dict-like object"
> returned by `MetaClass.__prepare__` is used as the `__dict__`
> for the forward-declared class object.
>
> (This is also novel.  Normally the "dict-like object" is
> used as the namespace for the class body, then its contents
> are copied out and it is discarded.  Here it will also be
> used as the `__dict__` for the forward-declared class object
> until the `continue class` statement executes.)
>
> Splitting `__new__` into two methods in this manner has several
> ramifications for existing code.
>
> First, Python still needs to support `MetaClass.__new__`
> for backwards compatibility with existing code.  Therefore,
> when executing the `class` statement, Python will still call
> `MetaClass.__new__`.  In fact, for maximum backwards
> compatibility, the order of externally-visible events
> for the `class` statement should not change at all.
>
> The default implementation of `MetaClass.__new__` will be
> changed to call `__new_forward__` and `__new_continue__`.
> The implementation will be similar to the following
> pseudo-code:
>
> ```Python
>     def __new__(metaclass, name, bases, namespace, **kwargs):
>         cls = metaclass.__new_forward__(metaclass, name, bases,
> namespace, **kwargs)
>         metaclass.__new_continue__(metaclass, cls, namespace, **kwargs)
>         return cls
> ```
>
> This means the order of events will be slightly different
> between a class defined with the `class` statement and
> a class defined with the `forward class` and `continue class`
> statements.  With a `class` statement, the class body will be
> run *before* `__new_forward__` is called, but with a `forward class`
> statement, the class body will be run *after* `__new_forward__`
> is called.  (This is why `__new_forward__` cannot know in advance
> whether or not the class body has been called, and the `namespace`
> has been filled in.)
>
> User code that defines its own metaclass with its own `__new__`
> must also continue to work.  But this leads us to a dangerous
> boundary condition:
>   * if user code defines a metaclass, and
>   * that metaclass defines `__new__` but not `__new_forward__` or
>     `__new_continue__`, and
>   * user code then uses that metaclass in a `forward class`
>     declaration, then
> Python must throw a `TypeError` exception.  This situation is
> unsafe: clearly the intention with the user's metaclass is to
> override some behavior in `__new__`, but the `forward new` statement
> will never call `__new__`.
>
> (It's safe to use a metaclass with `forward class` if it doesn't
> define `__new__`, or if it defines both `__new__` and either
> `__new_forward__` or `__new_continue__`.  It's also safe to
> use a metaclass with `class` if it defines either `__new_forward__`
> or `__new_continue__` but not `__new__`, because the default `__new__`
> will call both `__new_forward__` and `__new_continue__`.)
>
> Going forward, best practice for metaclasses would be to only
> implement `__new_forward__` and `__new_continue__`.
> Code with metaclasses that wanted to simultaneously support
> versions of Python with these new dunder methods *and* older
> versions of Python that predate this change would likely have
> to conditionally define their own `__new__`, best practices
> on this approach TBD.
>
> #### Interactions between `class`, `forward class`, and `continue class`
>
> `class` and `forward class` both bind a name to a newly-created object.
> Thus, in the same way that you can have two `class` statements that
> bind and re-bind the same name:
>
> ```Python
>     class C:
>         pass
>     class C:
>         pass
> ```
>
> You can execute `class` and `forward class` statements in any order
> to bind and re-bind the same name:
>
> ```Python
>     class C:
>         pass
>     forward class C
> ```
>
> This works as expected; when this code executes, the previous objects
> are dereferenced, and only the last definition of `C` is kept.
>
> Executing a `continue class` statement with a class defined by the
> `class` statement raises a `ValueError` exception.
> Executing a `continue class` statement with a class defined by the
> `forward class` statement that has already had `continue class`
> executed on it raises a `ValueError` exception.
>
> It's expected that knowledgeable users will be able to trick
> Python into executing `continue class` on the same class multiple
> times by interfering with "dunder" attributes.  The same tricks
> may also permit users to trick Python into executing `continue class`
> on a class defined by the `class` statement.  This is undefined
> and unsupported behavior, but Python will not prevent it.
>
>
> Final Notes
> -----------
>
> #### Alternate syntax
>
> Instead of `forward class`, we could use `def class`.  It's not as
> immediately clear, which is why this PEP prefers `forward class`.
> But this alternate syntax has the advantage of not adding a new
> keyword to the language.
>
>
> #### forward classes and PEP 649
>
> I suggest that `forward class` meshes nicely with PEP 649.
>
> PEP 649 solves the forward-reference and circular-reference
> problem for a lot of use cases, but not all.  So by itself
> it's not quite a complete solution to the problem.
>
> This `forward class` proposal should solve *all* the
> forward-reference and circular-reference problems faced
> by Python users today.  However, its use requires
> backwards-incompatible code changes to user code.
>
> By adding both PEP 649 and `forward class` to Python, we
> get the best of both worlds.  PEP 649 should handle most
> forward-reference and circular-reference problems, but the
> user could resort to `forward class` for the stubborn edge
> cases PEP 649 didn't handle.
>
> In particular, combining this PEP with PEP 649 achieves total
> coverage of the challenges cited by PEP 563's *Rationale* section:
>
> > * forward references: when a type hint contains names that have not
> been
> >   defined yet, that definition needs to be expressed as a string
> literal;
> > * type hints are executed at module import time, which is not
> >   computationally free.
>
> PEP 649 solves many forward reference problems, and delays the evaluation
> of annotations until they are used.  This PEP solves the remaining
> forward
> reference problems.
>
>
> ### A proof-of-concept using decorators
>
> I've published a repo with a proof-of-concept of
> the `forward class` / `continue class` syntax,
> implemented using decorators.
> It works surprisingly well, considering.
> You can find the repo here:
>
>     https://github.com/larryhastings/forward
>
> Naturally, the syntax using this decorators-based version
> can't be quite as clean.  The equivalent declaration for
> `class X` using these decorators would be as follows:
>
> ```Python
>     from forward import *
>
>     @forward()
>     class X():
>        ...
>
>     @continue_(X)
>     class _:
>        # class body goes here
>        pass
> ```
>
> Specifically:
>
> * You have to make the `forward` module available somehow.  You can
> just copy the
>   `forward` directory into the directory you want to experiment in, or
> you can
>   install it locally in your Python install or venv by installing the
> `flit`
>   package from PyPI and running `flit install -s` .
> * You must import and use the two decorators from the `forward` module.
>   The easiest way is with `from forward import *` .
> * For the `forward class` statement, you instead decorate a
> conventional class
>   declaration with `@forward()`.  The class body should be empty, with
> either
>   a single `pass` statement or a single ellipsis `...` on a line by
> itself;
>   the ellipsis form is preferred.  You should name this class with the
> desired
>   final name of your class.
> * For the `continue class` statement, you instead decorate a conventional
>   class declaration with `@continue_()`, passing in the
> forward-declared class
>   object as a parameter to the decorator.  You can use the original
> name of
>   the class if you wish, or a throwaway name like `_` as per the example.
> * You may use additional decorators with either or both of these
> decorators.
>   However it's vital that `@forward()` and `@continue_()` are the
>   *first* decorators run--that is, they should be on the *bottom* of the
>   stack of decorators.
>
> Notes and caveats on the proof-of-concept:
>
> * The `continue_` decorator returns the original "forwarded" class
> object.
>   This is what permits you to stack additional decorators on the class.
>   (But, again, you *must* call the `continue_` decorator first--it should
>   be on the bottom.)
> * To use `__slots__`, you will have to declare them in the `forward`
> class.
> * The proof-of-concept can't support classes that inherit from a class
>   which defines `__init_subclass__`.
> * Like the proposed syntax, this proof-of-concept doesn't support
>   decorators that both examine the contents of the class *and* return
>   a different object, e.g. `@dataclass(slots=True)` in Python 3.10.
> * This proof-of-concept doesn't work with metaclasses that
>   override either `__new__` or `__init__`, where those functions
>   examine the `namespace` argument in any meaningful way.
>
>
> #### tools/
>
> There are some tools in the `tools/` directory that will (attempt to)
> automatically add or remove the `@forward()` decorator to class
> definitions
> in Python scripts.  It turns this:
>
> ```Python
>     class foo(...):
>         pass
> ```
>
> into this:
>
> ```Python
>     @forward()
>     class foo(...):
>         ...
>
>     @continue_(foo)
>     class _____:
>         pass
> ```
>
> `tools/edit_file.py` will edit one or more Python files specified on the
> command-line, making the above change.  By default it will toggle the
> presence of `@forward`
> decorators.  You can also specify explicit behavior:
>
> `-a` adds `@forward()` decorators to `class` statements that don't
> have them.
> `-r` removes `@forward` decorators, changing back to conventional
> `class` statements.
> `-t` requests that it "toggle" the state of `@forward()` decorators.
>
> The parser is pretty dumb, so don't run it on anything precious. If it
> goofs up, sorry!
>
> `tools/edit_tree.py` applies `edit_py.py` to all `*.py` files found
> anywhere under
> a particular directory.
>
> `tools/edit_stdlib.py` was an attempt to intelligently apply
> `edit_file.py` to the `Lib`
> tree of a CPython checkout.  Sadly, the experiment didn't really work
> out; it seemed
> like there were so many exceptions where the brute-force modification
> didn't work,
> either due to descriptors, metaclasses, or base classes with
> `__init_subclass__`,
> that I gave up on the time investment.  It's provided here in a
> non-functional
> state in case anyone wants to experiment with it further.
>
> Also, it's intentionally delicate; it only works on git checkout
> trees, and only with
> one specific revision id:
>
>     7b87e8af0cb8df0d76e8ab18a9b12affb4526103
>
> #### Postscript
>
> Thanks to Eric V. Smith and Barry Warsaw for proofreading and ideas.
> Thanks in particular to Eric V. Smith for the idea about making
> `__slots__` processing lazy.
> Thanks to Mark Shannon for the idea of prototyping `forward class`
> and `continue class` using decorators and simply copying the
> attributes.
>
>
> _______________________________________________
> Python-Dev mailing list -- pytho...@python.org
> To unsubscribe send an email to python-d...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/pytho...@python.org/message/CW6Z3OS42DYNMO4Y42R6O3LVTPPAA57X/
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/WSJGXGVOIWUAQWEJ2GAAAL3QT2BJ46K3/

Paul Moore

unread,
Apr 23, 2022, 6:13:25 PM4/23/22