[Python-ideas] Allow Enum members to refer to each other during execution of body

[Antony Lee]

Currently, during the execution of the body of the Enum declaration, member names are bound to the values, not to the Enum members themselves.  For example

class StateMachine(Enum):
    A = {}
    B = {1: A} # e.g. a transition table

StateMachine.B[1] == {}, when one could have expected StateMachine.B[1] == StateMachine.A

It seems to me that a behavior where member names are bound to the members instead of being bound to the values is more useful, as one can easily retrieve the values from the members but not the other way round (at least during the execution of class body).

Initially, I thought that this could be changed by modifying _EnumDict, so that its __setitem__ method sets the member in the dict, instead of the value, but in fact this doesn't work because while the values are being set in the _EnumDict the class itself doesn't exist yet (and for good reason: the __init__ and __new__ methods may be defined later but there is no way to know that).  However, a possible solution could to momentarily create Enum members as instances of some dummy class, and then later, after execution of class body has completed, change the members' class to the actual Enum and initialize them as needed (if an __init__ or a __new__ are actually defined).  Well, there are limitations with this approach (e.g. the members are not fully initialized before class body finishes to execute) but this seems better than the current behavior(?)

Best,

Antony

[Ethan Furman]

Part of the problem here would be maintaining the linkage when the temp enum object from _EnumDict was translated into
an actual Enum member.

One possible work around is to store the name of the member instead:

class StateMachine(Enum):
A = {}
B = {1:'A'}

then the other methods can either dereference the name with an __getitem__ look-up, or the class can be post-processed
with a decorator to change the strings back to actual members... hmmm, maybe a post_process hook in the metaclass would
make sense?

--
~Ethan~
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas

[Haoyi Li]

> then the other methods can either dereference the name with an __getitem__ look-up, or the class can be post-processed with a decorator to change the strings back to actual members... hmmm, maybe a post_process hook in the metaclass would make sense?

Having real strings be part of the enums data members is a pretty common thing, and working through and trying to identify the linkage-strings from normal-strings seems very magical to me. Is there some metaclass-magic way to intercept the usage of A, to instead put the enum instance there?

Also, for this to be useful for your described use case, (state machines yay!) you'd probably want to be able to define back/circular references, which i think isn't currently possible. The obvious thing to do would be to somehow make the RHS of the assignments lazy, which would allow out-of-order and circular assignments with a very nice, unambigious:

class StateMachine(Enum):

    "Useless ping-pong state machine"
    A = {1: B}
    B = {1: A}

But short of using macros to do an AST transform, I don't know if such a thing is possible at all.

-Haoyi

[Ethan Furman]

On 07/08/2013 06:03 PM, Haoyi Li wrote:

On 07/08/2013 Ethan Furman wrote:
>>
>> then the other methods can either dereference the name with an __getitem__
>> look-up, or the class can be post-processed with a decorator to change the
>> strings back to actual members... hmmm, maybe a post_process hook in the
>> metaclass would make sense?
>
> Having real strings be part of the enums data members is a pretty common
> thing, and working through and trying to identify the linkage-strings from
> normal-strings seems very magical to me. Is there some metaclass-magic way
> to intercept the usage of A, to instead put the enum instance there?

The post-processing routines would be specific to this enumeration, so they
should know which strings are enum references and which aren't.

> Also, for this to be useful for your described use case, (state machines yay!) you'd probably want to be able to define
> back/circular references, which i think isn't currently possible. The obvious thing to do would be to somehow make the
> RHS of the assignments lazy, which would allow out-of-order and circular assignments with a very nice, unambigious:
>
> class StateMachine(Enum):
> "Useless ping-pong state machine"
> A = {1: B}
> B = {1: A}
>
> But short of using macros to do an AST transform, I don't know if such a thing is possible at all.

Starting off with strings and post-processing would handle it nicely.

[Antony Lee]

"Part of the problem here would be maintaining the linkage when the temp enum object from _EnumDict was translated into an actual Enum member."

I implemented the required behavior here: https://github.com/anntzer/enum

Instead of creating a new enum object from the temp object stored in _EnumDict, I directly change the class of the temp object to its actual value once that class is built, thus keeping references correct (see test_backward_reference).  However, this behavior breaks down if the Enum class also inherits from a type with a different layout (e.g., int), because I can't change the class of such objects.  In fact, even

class A(int): pass
class B(int): pass
A().__class__ = B

fails (which is puzzling to me... I understand that you can't transform an instance of an int-subclass into an instance of an str-subclass), but here both classes should have the same layout...

On the other hand IntEnums shouldn't need that kind of behavior anyways, so I just kept the old implementation for them (for any class for which instances can't be instantiated by object.__new__(cls), in fact).

I haven't worked on forward-references but this should be not too hard to implement either: just add a __missing__ to _EnumDict (cf. the discussion on implicit enums) that creates temporary placeholder members on the fly.  When these members are actually defined, initialize them.  When class body finishes to execute, check that all placeholders have been initialized, throwing an error otherwise.

Antony

2013/7/8 Antony Lee <anton...@berkeley.edu>

[Antony Lee]

Forward references are now implemented (https://github.com/anntzer/enum).  They require an explicit declaration, à la

class C(Enum, declaration=...):
    B = ...

    A = {1: B}
    B = {1: A}

I had implemented a version where the initial declaration wasn't needed, but as mentioned in previous enum-related threads this can create many problems.  For example, consider

class C(Enum):

    A = {1: B}; B = {1: A}

    @property
    def also_value(self): return self.value

how is Python supposed to know that when it tries to resolve "B" in the class dict, it must create a new member, but when it tries to resolve "property" in the class dict and doesn't find it, it must look in the enclosing scope?  You can decide that a name lookup creates a new member if the name isn't defined in the enclosing scope either (I implemented this using sys._getframe in a previous commit of my fork) but this leads to other (somewhat contrieved) problems:

x = 1
class C(Enum):
    y = x # <- what is this supposed to mean?
    x = 2

Note that even AST macros don't (fully) this issue because you can't really even know the list of all names that are defined in the class body:

x = 1
def inject(**kwargs):
    for k, v in kwargs.items(): sys._getframe(1).f_locals[k] = v # interestingly using dict.update does not trigger the use-defined __setitem__
class C(Enum):
    y=x # <- ???
    inject_value(x=2)

Antony

[Ethan Furman]

On 07/10/2013 03:47 PM, Antony Lee wrote:
>
> Forward references are now implemented (https://github.com/anntzer/enum). They require an explicit declaration, à la

Do they work with a custom __new__ ? __init__ ?

[Antony Lee]

In the current version, they work with a custom __init__ (though of course, as long as the actual arguments that need to be passed to __init__ are provided, the pre-declared members are just "empty").  They do not work with a custom __new__ (not sure how I could make this work, given that at declaration time an "empty" member needs to be created but we don't know what arguments we need to pass to __new__...).
As a side effect, however, the whole patch adds a new requirement: custom __new__s must be defined before the members themselves; otherwise they won't be called, for the same reason as above: if I don't know what __new__ is, I can't call it...

Antony

[Ethan Furman]

On 07/11/2013 02:07 PM, Antony Lee wrote:
> On Wednesday, July 10, 2013 6:52:24 PM UTC-7, stoneleaf wrote:
>> On 07/10/2013 03:47 PM, Antony Lee wrote:
>>>

>>> Forward references are now implemented (https://github.com/anntzer/enum <https://github.com/anntzer/enum>).

>>
>> Do they work with a custom __new__ ? __init__ ?
>

> In the current version, they work with a custom __init__ (though of course, as long as the actual arguments that need to
> be passed to __init__ are provided, the pre-declared members are just "empty"). They do not work with a custom __new__
> (not sure how I could make this work, given that at declaration time an "empty" member needs to be created but we don't
> know what arguments we need to pass to __new__...).
> As a side effect, however, the whole patch adds a new requirement: custom __new__s must be defined before the members
> themselves; otherwise they won't be called, for the same reason as above: if I don't know what __new__ is, I can't call
> it...

Hmm. Well, at this point I can offer kudos for getting it this far, but that's about it. The use-case this addresses
seems fairly rare, and is definitely not a typical enumeration, and can be solved fairly easily with some extra
post-processing code on a per-enumeration basis.

[Antony Lee]

Is there any specific reason why you do not wish to change the behavior of Enum to this one (which does seem more logical to me)?  The patch is fairly simple in its logic (compared to the rest of the implementation, at least...), and I could even change it to remove the requirement of defining __new__ before the members as long as there are no references to other members (because as long as there are no references to other members, I obviously don't need to actually create the members), thus making it fully compatible with the current version.
Antony

2013/7/11 Ethan Furman <et...@stoneleaf.us>

--

--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/PC_Ej19qj5w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Antony Lee]

Sorry for the duplicate, seems like sending just to the googlegroups address doesn't cc to the python.org address.
My previous email below.

Antony

2013/7/12 Antony Lee <anton...@berkeley.edu>

[Ethan Furman]

On 07/14/2013 02:36 PM, Antony Lee wrote:
> Is there any specific reason why you do not wish to change the behavior of Enum to this one (which does seem more
> logical to me)? The patch is fairly simple in its logic (compared to the rest of the implementation, at least...),
> and I could even change it to remove the requirement of defining __new__ before the members as long as there are no
> references to other members (because as long as there are no references to other members, I obviously don't need to
> actually create the members), thus making it fully compatible with the current version.

My apologies for the delay in replying.

Getting Enum into the stdlib was a very careful balancing act:

- Make it powerful enough to meet most needs as-is

- Make it extensible enough that custom enumerations could be
easily implemented

- Make it simple enough to not create a large cognitive burden

How this relates to your patch:

1) With your patch, referencing another enum member either returns
the member itself (pure Enum), or the value of the Enum (mixed
Enum) -- which means two different behaviors from the same
syntax.

2) The patch fails with the pure Enum with auto-numbering test case.
It fails because __new__ is looking at the __member__ data
structure which is empty for the duration of __prepare__. While
work arounds are possible, they would not be simpler, or even as
simple.

Summary: The resulting behavior is inconsistent, and the complexity added to the code, but mostly to the mind, is much
greater than the minor benefit.

--
~Ethna

[Antony Lee]

2013/7/19 Ethan Furman <et...@stoneleaf.us>

On 07/14/2013 02:36 PM, Antony Lee wrote:

Is there any specific reason why you do not wish to change the behavior of Enum to this one (which does seem more
logical to me)?  The patch is fairly simple in its logic (compared to the rest of the implementation, at least...),
and I could even change it to remove the requirement of defining __new__ before the members as long as there are no
references to other members (because as long as there are no references to other members, I obviously don't need to
actually create the members), thus making it fully compatible with the current version.

My apologies for the delay in replying.

Getting Enum into the stdlib was a very careful balancing act:

    - Make it powerful enough to meet most needs as-is

    - Make it extensible enough that custom enumerations could be
      easily implemented

    - Make it simple enough to not create a large cognitive burden

Sure, I understand that.  I would like to argue that the patch is in fact not as complex as you claim, though obviously at the end I can't make any decision.  Basically, my point of view is that

class C(Enum):
   <code>

should be the same as executing <code> in a new namespace, with the ONLY exception that each assignment (of a value that isn't a descriptor) to the namespace instead creates an enum member, either by Enum.__new__/__init__, or by whatever __new__/__init__ had been defined earlier, and assigns that member to the namespace.

How this relates to your patch:

1)  With your patch, referencing another enum member either returns
    the member itself (pure Enum), or the value of the Enum (mixed
    Enum) -- which means two different behaviors from the same
    syntax.

While this is not implemented, I can (probably...) make sure that referencing another enum member always either returns the member itself, if available, or raises an error if not; the value of the Enum would always be available as member.value.

2)  The patch fails with the pure Enum with auto-numbering test case.
    It fails because __new__ is looking at the __member__ data
    structure which is empty for the duration of __prepare__.  While
    work arounds are possible, they would not be simpler, or even as
    simple.

This is not the case: test_enum.TestEnum.test_duplicate_values_give_unique_enum_items still passes, as long as __new__ is defined ahead of the members.

Summary:  The resulting behavior is inconsistent, and the complexity added to the code, but mostly to the mind, is much greater than the minor benefit.

--
~Ethna

_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas