I'm unsure about the forward class. How is it different from subclassing an ABC?
They're just different objects. A subclass of an ABC is either
itself another abstract base class, which will never be
instantiatable, or a non-abstract class, which is immediately
instantiatable. A forward-declared class object is not currently
instantiatable, and is not fully defined, but will become fully
defined and instantiatable after the matching "continue class"
statement.
What happens if you try to continue a non-forward class?
From the proto-PEP:
Executing a `continue class` statement with a class defined by the `class` statement raises a `ValueError` exception.
And also:
It's expected that knowledgeable users will be able to trick Python into executing `continue class` on the same class multiple times by interfering with "dunder" attributes. The same tricks may also permit users to trick Python into executing `continue class` on a class defined by the `class` statement. This is undefined and unsupported behavior, but Python will not prevent it.
/arry
On 4/22/2022 9:13 PM, Larry Hastings wrote:
forward class X()
New keywords are a nuisance. And the proposed implementation seems too complex.
My proposed implementation seemed necessary to handle the
complexity of the problem. I would welcome a simpler solution
that also worked for all the same use cases.
How about a 'regular' class statement with a special marker of some sort. Example: 'body=None'.
It's plausible. I take it "body=None" would mean the declaration would not be permitted to have a colon and a class body. So now we have two forms of the "class" statement, and which syntax you're using is controlled by a named parameter argument.
I think this "body=None" argument changing the syntax of the "class" statement is clumsy. It lacks the visibility and clarity of "forward class"; the keyword here makes it pretty obvious that this is not a conventional class declaration. So I still prefer "forward class".
In my PEP I proposed an alternate syntax for "forward class":
"def class", which has the feature that it doesn't require adding
a new keyword. But again, I don't think it's as clear as "forward
class", and I think clarity is vital here.
Either __new__ or __init__ could raise XError("Cannot instantiate until this is continued.", so no special instantiation code would needed and X could be a real class, with a special limitation.
Yes, my proposal already suggests that __new__ raise an exception. That's not the hard part of the problem.
The problem with X being a "real class" is that creating a "real
class" means running all the class creation code, and a lot of the
class creation code is designed with the assumption that the
namespace has already been filled by executing the class body.
For example, Enum in enum.py relies on EnumMeta, which defines
__new__, which examines the already-initialized namespace of the
class you want to create. If you propose a "body=None" class be a
"real class object", then how do you declare a class that inherits
from Enum using "body=None"? Creating the class object will call
EnumMeta.__new__, which needs to examine the namespace, which
hasn't been initialized yet.
Changing the class object creation code so we can construct a class object in two steps, with the execution of the class body being part of the second step, was the major sticking point--and the source of most of the complexity of my proposal.
continue class X:
# class body goes here
def __init__(self, key):
self.key = key
'continue' is already a keyword.
I'm aware. I'm not sure why you mentioned it.
Given that X is a real class, could implementation be X.__dict__.update(new-body-dict)
That's what the proof-of-concept does. But the proof-of-concept fails with a lot of common use cases:
because these methods assume the namespace of the class has
already been filled in, but it doesn't get filled in until the
@continue_() class decorator. Handling these cases is why my
proposal is sadly as complex as it is, and why in practice the
proof-of-concept doesn't work a lot of the time.
/arry
On Fri, Apr 22, 2022 at 06:13:57PM -0700, Larry Hastings wrote:This PEP proposes an additional syntax for declaring a class which splits this work across two statements: * The first statement is `forward class`, which declares the class and binds the class object. * The second statement is `continue class`, which defines the contents of the class in the "class body". To be clear: `forward class` creates the official, actual class object. Code that wants to take a reference to the class object may take references to the `forward class` declared class, and interact with it as normal. However, a class created by `forward class` can't be *instantiated* until after the matching `continue class` statement finishes.Since the "forward class" is a real class,
It's a "forward-declared class object". It's the real class
object, but it hasn't been fully initialized yet, and won't be
until the "continue class" statement.
it doesn't need any new syntax to create it. Just use plain old regular class syntax. class X(object): """Doc string""" attribute = 42 And now we have our X class, ready to use in annotations. To add to it, or "continue class" in your terms, we can already do this: X.value = "Hello World"
But if X has a metaclass that defines __new__ , "value" won't be defined yet, so metaclass.__new__ won't be able to react to and possibly modify it. Similarly for metaclass.__init__ and BaseClass.__init_subclass__.
So, while your suggested technique doesn't "break" class creation
per se, it prevents the user from benefiting from metaclasses and
base classes using these advanced techniques.
Counter proposal: `continue class expression:` evaluates expression to an existing class object (or raises an exception) and introduces a block. The block is executed inside that class' namespace, as the `class` keyword does, except the class already exists.
If "continue class" is run on an already-created class, this
breaks the functionality of __prepare__, which creates the
namespace used during class body execution and is thrown away
afterwards. The "dict-like object" returned by __prepare__ will
have been thrown away by the time "continue class" is executed.
Also, again, this means that the contents added to the class in
the "continue class" block won't be visible to metaclass.__new__,
metaclass.__init__, and BaseClass.__init_subclass__.
Also, we would want some way of preventing the user from running
"continue class" multiple times on the same class--else we
accidentally condone monkey-patching in Python, which we don't
want to do.
Isn't this case solved by either forward references: class A: value: "B" or by either of PEP 563 or PEP 649?
It is, but:
a) manual stringizing was rejected by the community as too
tiresome and too error-prone (the syntax of the string isn't
checked until you run your static type analysis tool). Also, if
you need the actual Python value at runtime, you need to eval()
it, which causes a lot of headaches.
b) PEP 649 doesn't solve this only-slightly-more-advanced case:
@dataclass
class A:
value: B
@dataclass
class B:
value: A
as the dataclass decorator examines the contents of the class, including its annotations.
c) PEP 563 has the same "what if you need the actual Python value at runtime" problem as manual stringization, which I believe is why the SC has delayed its becoming default behavior.
Perhaps my example for b) would be a better example for the PEP.
That could become: class A: pass class B: value: A # This is fine, A exists. A.value: B # And here B exists, so this is fine too. No new syntax is needed. This is already legal.
It's legal, but it doesn't set the annotation of "value" on A.
Perhaps this is just a bug and could be fixed. (TBH I'm not sure
what the intended semantics of this statement are, or where that
annotation ends up currently. I couldn't find it in
A.__annotations__ or the module's __annotations__. Is it just
forgotten?)
I assert this approach will be undesirable to Python programmers. This makes for two very-different feeling approaches to defining the members of a class. One of the goals of my PEP was to preserve the existing "feel" of Python as much as possible.
Also, as previously mentioned, your technique prevents "A.value", and all other attributes and methods set using this technique, from being visible to metaclass.__new__, metaclass.__init__, and BaseClass.__init_subclass__.
This proposed `forward class` / `continue class` syntax should permit solving *every* forward-reference and circular-reference problem faced in Python,I think that's overselling the concept.
Okay, perhaps I should have said "the forward-reference and
circular-reference problems of class definitions" or something
like that. I'll adjust the text for the second draft.
using an elegant and Pythonic new syntax.That's a matter of opinion.
Yes. Are PEPs not permitted to express opinions?
As a side benefit, `forward class` and `continue class` syntax enables rudimentary separation of "interface" from "implementation", at least for classes.I don't think so. The `forward class` syntax doesn't define any part of the interface except the class' name.
It also defines the base classes and metaclass, as well as some other simple metadata ("__file__"), all which may be of interest to external consumers of the object.
If you don't like the PEP calling this 'the rudimentary
separation of 'interface' from 'implementation'", what is your
counter-proposal?
However, the user isn't permitted to instantiate a forward-declared class object until after the corresponding `continue class X`.Why not?
Because the class object is not fully initialized yet; it's a
"forward-declared class object". I thought my PEP was pretty
clear on that point.
Feel free to make your counter-proposal, but it's incoherent to
debate the statements of my proposal as if they're statements
about your counter-proposal.
Since you explicitly allow the user to instantiate the class by first removing the `__forward__` dunder,
"allowing" and "condoning" are two different things. Python
allows a lot of things that are not condoned.
/arry
Anyhow, [a forward-defined class object is] a class, with some special features (notably that you can't instantiate it).
Yes. Specifically, here's my intention for "forward-defined
class objects": you can examine some generic dunder values
(__name__, __mro__), and you can take references to it. You can't
instantiate it or meaningfully examine its contents, because it
hasn't been fully initialized yet.
It seems odd that you define a blessed way of monkeypatching a class, but then demand that it can only be done once unless you mess with dunders. Why not just allow multiple continuations?
I think monkeypatching is bad, and I'm trying to avoid Python condoning it.
On that note, the intent of my proposal is that "continue class"
is not viewed as "monkeypatching" the class, it's the second step
in defining the class.
I considered attempting to prevent the user modifying the "forward-declared class object". But a) that just seemed like an arms race with the user--"oh yeah? well watch THIS!" and b) I thought the Consenting Adults rule applied.
Still, it's not the intent of my PEP to condone or facilitate
monkeypatching.
/arry
My main question for this approach is how would this work with type checkers?
It would be new syntax for Python, so type checkers would have to understand it.
Is there any restriction that forward class's continuation must appear in same module?
No.
If it's allowed that a forward class may be continued in a different module I do not see how type checker like mypy/pyright could handle that. Classes are generally viewed as closed and fully defined within type checker. Monkey patching at runtime later is not supported.
If it became official Python syntax, I suspect they'd figure out a way to support it.
They might require that the expression used in the "continue
class" statement map to the original "forward class" declaration,
e.g. they might stipulate that they don't support this:
forward class X
random_name = X
continue class random_name:
...
But rather than speculate further, perhaps someone who works on
one of the static type analysis checkers will join the discussion
and render an informed opinion about how easy or hard it would be
to support "forward class" and "continue class".
One other edge case here is how would you forward declare an annotation for a type that like this, if TYPE_CHECKING: import numpy def f(x: numpy.ndarray) -> None: ... forward declaring ndarray here would not make numpy.ndarray available.
In this case, adding forward declarations for classes in "numpy" would be up to the "numpy" module. One approach might look more like this:
import numpy # contains forward declarations
if TYPE_CHECKING:
import numpy.impl
Though numpy presumably couldn't do this while they still
supported older versions of Python. That's one downside of using
new syntax--you can't use it until you stop support for old
versions of Python that predate it.
Would you forward declare modules? Is that allowed?
I haven't proposed any syntax for forward-declaring modules, only
classes.
I'm confused in general how if TYPE_CHECKING issue is handled by this approach. Usually class being imported in those blocks is defined normally (without continue class) somewhere else.
My proposal should mate well with "if TYPE_CHECKING". You would
define your forward classes in a module that does get imported,
but leave the continue classes in a separate module that is only
imported "if TYPE_CHECKING", as per my example with "numpy" above.
/arry
It's a "forward-declared class object". It's the real class object, but it hasn't been fully initialized yet, and won't be until the "continue class" statement.The only thing that makes it not fully initialised is that it has a bozo bit dunder "__forward__" instructing the interpreter to disallow instantiation. Yes? If I take that class object created by `forward class X`, and delete the dunder, there is no difference between it and other regular classes. Am I correct?
No, there are several differences.
The "forward-declared class object" is in a not-yet-fully initialized state, and is not ready for use as a class.
From my perspective, the "__forward__" attribute is an internal
implementation detail, and something that user code should
strictly leave alone. But if it's considered too dangerous to
expose to users, we could hide it in the class object and not
expose it to users. I'm not convinced that's the right call; I
think the Consenting Adults rule still applies. Python lets you
do crazy things like assigning to __class__, and resurrecting
objects from inside their __del__; manually removing __forward__
seems like it falls into the same category. It's not recommended,
and we might go so far as to say doing that results in undefined
behavior. But Python shouldn't stand in your way if you really
think you need to do it for some reason.
/arry
On Sat, Apr 23, 2022 at 12:46:37AM -0700, Larry Hastings wrote:But rather than speculate further, perhaps someone who works on one of the static type analysis checkers will join the discussion and render an informed opinion about how easy or hard it would be to support "forward class" and "continue class".No offense Larry, but since this proto-PEP is designed to help the typing community (I guess...) shouldn't you have done that before approaching Python-Dev with the proposal?
The perfect is the enemy of the good. Like I said, I wanted to
get this out there before the Language Summit, and I just ran out
of time. I think there's also some sort of typing summit next
week? I'm not really plugged in to the static type analysis
world--I don't use it in any of my projects.
Wouldn't that be a massively breaking change? Anyone who does: from numpy import ndarray will get the forward-declared class object instead of the fully initialised class object, leading to all sorts of action-at-a-distance bugs.
I wasn't recommending The Famous numpy Project do this exact
thing, it was an abstract example using the name "numpy". I
didn't think this was a real example anyway, as I was assuming
that most people who import numpy don't do so in an "if
TYPE_CHECKING:" block.
Separating the forward class declaration from the continue class implementation in the actual "numpy" module itself is probably not in the cards for a while, if ever. But perhaps numpy could do this:
import numpy.forward
if TYPE_CHECKING:
import numpy
In this case, the "numpy" module would also internally "import numpy.forward", and would contain the "continue class" statements for the forward-declared classes in "numpy.forward".
There are lots of ways to solve problems with the flexibility
afforded by the proposed "forward class" / "continue class"
syntax. Perhaps in the future you'll suggest some of them!
/arry
On 4/22/2022 11:16 PM, Larry Hastings wrote:
So I still prefer "forward class".
I don't think it's as clear as "forward class"
'forward class' for an incomplete class is not at all clear to me. It is not clear to me which part of speech you intend it to be: noun, verb, adjective, or adverb. You must have some experience with 'forward' in a different context that makes it clearer to you.
It's a reference to the term "forward declaration":
/arry
But the explosion of static type analysis in Python, particularly with
the `typing` module and the `mypy` tool, has made circular definition-time
dependencies between classes commonplace--and much harder to solve. Here's
one simple example:
```Python
class A:
value: B
class B:
value: A
```
However, it doesn't solve the problem for base classes. For example, str is conceptually defined as `class str(Sequence["str"]):`. A forward reference can't make `str` defined when the bases are evaluated, because bases are resolved at the `forward class` stage.
Larry's second email "Proto-PEP part 2: Alternate implementation
proposal for "forward class" using a proxy object" discusses a
possibility to move the bases and metaclasses to the "continue
class" stage. It also has the advantage of not changing the
behavior of __new__, and I think is in general easier to reason
about. He and I have discussed this approach, but neither of have
looked at in enough detail to know if the implementation is
possible. Some of the concerns are noted in that email.
Eric
On the other hand, there's something I've been seeing around. I don't know if it was introduced by Mypy or something, but its the use of declaration files. I think they are saved as pyi files. They just have the declaration of a python object, be it class or variable. What if we just found a way of reusing that instead?
As they currently exist, stub files (.pyi files) don't contain enough information. In particular, they don't have the metaclass information. This could be changed, but at that point you basically have the "forward" declaration, but it's hidden away where the interpreter can't see it.
Eric
So to reiterate, your proposal would be to write this as:
forward class B:pass
class A:value: B
continue class B:value: A
Not quite; the "forward class" statement doesn't have a colon or a class body. This would be written as:
forward class B
class A:
value: B
continue class B:
value: A
While the current workaround is:
class A:value: "B"
class B:value: "A"
In this example, with two toy classes in one file, it shouldn't be necessary to quote the annotation in B. So all you need is the quotes around the first annotation:
class A:
value: "B"
class B:
value: A
I don't think I would write the "forward class" version if I had the choice. It's clunkier and requires more refactoring if I change my mind about whether the `value` attribute should exist.
In this toy example, it adds an extra line. Describing that as "clunky" is a matter of opinion; I disagree and think it's fine.
But the real difference is when it comes to larger codebases. If
classes "A" and "B" are referenced dozens or even hundreds of
times, you'd have to add quote marks around every annotation that
references one (both?). Manual stringization of large codebases
was sufficiently disliked as to have brought about the creation
and acceptance of PEP 563. Judicious use of the "forward class"
statement should obviate most (all?) the manual stringizing in
these codebases.
/arry
On 4/23/2022 9:55 AM, Jelle Zijlstra wrote:
However, it doesn't solve the problem for base classes. For example, str is conceptually defined as `class str(Sequence["str"]):`. A forward reference can't make `str` defined when the bases are evaluated, because bases are resolved at the `forward class` stage.Larry's second email "Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object" discusses a possibility to move the bases and metaclasses to the "continue class" stage. It also has the advantage of not changing the behavior of __new__, and I think is in general easier to reason about.
Let me expound on Eric's statement for a bit. Moving the base
classes and metaclass to the "continue" statement might permit the
self-referential "str" definition suggested above:
forward class str
...
continue class str(Sequence[str]):
...
Though I suspect this isn't viable, or at least not today. I'm willing to bet a self-referential definition like this would result in an infinite loop when calculating the MRO.
I don't have a strong sense of whether it'd be better to have the base classes and metaclass defined with the "forward" declaration or the "continue" declaration. The analogous syntax in C++ has the base classes defined with the "continue" class.
Actually, that reminds me of something I should have mentioned in the proto-PEP. If the "class proxy" version is viable and desirable, and we considered moving the base classes and metaclass down to the "continue" statement, that theoretically means we could drop the "forward" and "continue" keywords entirely. I prefer them, simply because it makes it so explicit what you're reading. But the equivalent syntax in C++ doesn't bother with extra keywords for either the "forward" declaration or the "continue" declaration, and people seem to like it fine. Using that variant of the syntax, the toy example from the PEP would read as follows:
class A
class B:
value: A
class A:
value: B
If you're speed-reading here and wondering "wait, how is this not
ambiguous?", note that in the first line of the example, the
"class" statement has no colon. Also, it would never have
parentheses or decorators. The forward declaration of a class
would always be just "class", followed by a name, followed by a
newline (or comment).
/arry