Python project delays changes to annotations

[David Szent-Györgyi]

This was published a couple of weeks ago, on 9th June, but it is likely to be current as of this date (26th June). It is of interest to Leo's developers and other programmers who annotate their Python code with typing information: 

The following excerpt begins "When and how to evaluate Python annotations". A careful reading of the entire article and the comments posted in reply is needed. If one uses a Python package that depends on annotations, especially one that uses typing information or provides that,  a check with users of and developers of that package is in order. 

"Annotations in Python came late to the party; they were introduced in Python 3 as a way to attach information to functions describing their arguments and return values. While that mechanism had obvious applications for adding type information to Python functions, standardized interpretations for the annotations came later with type hints. But evaluating the annotations at function-definition time caused some difficulties, especially with respect to forward references to type names, so a Python Enhancement Proposal (PEP) was created to postpone their evaluation until they were needed. The PEP-described behavior was set to become the default in the upcoming Python 3.10 release, but that is not to be; the postponement of evaluation by default has itself been postponed in the hopes of unwinding things."

[David Szent-Györgyi]

From the article: 

"a Python Enhancement Proposal (PEP) was created to postpone their evaluation until they were needed. The PEP-described behavior was set to become the default in the upcoming Python 3.10 release, but that is not to be; the postponement of evaluation by default has itself been postponed in the hopes of unwinding things."

The present objection to PEP 649 is its breaking change to the __annotations__  attribute, since that attribute has been around since Python 3.0 and its use not restricted to types. Perhaps adding two attributes to objects would work: one would be the __co_annotations__ attribute described in PEP 649, the other would be a typing-specific attribute with use limited to typing, that would for that use take the place of the existing __annotations__ attribute.

This would require work by the authors of code that relies on the current __annotations__ attribute for typing-related information, but that may be the only sure answer that does not break code that uses the __annotations__ attribute for other purposes. 

[Edward K. Ream]

On Sat, Jun 26, 2021 at 7:41 AM David Szent-Györgyi <das...@gmail.com> wrote:

"When and how to evaluate Python annotations".

Thanks for the link. Your post is timely.  In the last few days, I have gone from a mypy newbie to a mypy journeyman.  Just today, in the ekr-annotations branch, mypy passes the fully annotated version of leoNodes.py.

The following statement is almost completely wrong:

"evaluating these annotations requires computation, so all programs pay the price of the annotations, even if they are never needed."

Annotations are pretty much the same as comments. They have no significant cost.

The following is a real problem:

" forward references to types that have not yet been defined requires using string literals, instead of type name".

The problem is dealing with the forward reference itself.  Imo, the ugliness of the string itself isn't significant.

The typing module defines the TYPE_CHECKING constant, which is always False at runtime. mypy, and other checkers, set this constant to True while doing analysis.

At present, the following appears at the start of leoGlobals.py:

# Do the following imports *only* when running mypy.
if TYPE_CHECKING:  # Always False at runtime.
    import leo.core.leoCommands as leoCommands
    import leo.core.leoNodes as leoNodes
    assert leoCommands
    assert leoNodes

Yes, this is ugly, but it allows string annotations like this:

def findTabWidthDirectives(

    c:"leoCommands.Commands", p:"leoNodes.Position"):

Summary

Imo, annotations will become an important part of Leo. mypy isn't perfect, but it is flexible and easy to use. I plan to fully annotate only Leo's crown jewels (the modules corresponding to c, g, and p), namely the leoCommands, leoGlobals, and leoNodes modules. mypy has already found about a half dozen bugs that pylint and Leo's unit tests have missed.

Leo will fail to start if leoGlobals.py imports any other Leo module. leoGlobals.py can deal with forward references, as shown above.

leoNodes.py also uses forward references to the Position and Vnode class, but those references don't require the TYPE_CHECKING constant the referenced classes appear in leoNodes.py.

I care little or nothing what Python 3.10 may bring. Imo, mypy and python's typing module seem sufficient just as they are.

Edward

[tbp1...@gmail.com]

With the complexity of Leo, I think type annotations will be a Good Thing.  I am a little concerned that when the typing issue settles down in future Python releases, the syntax may change enough that they would have to be redone.

[Edward K. Ream]

On Sat, Jun 26, 2021 at 6:16 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

With the complexity of Leo, I think type annotations will be a Good Thing.  I am a little concerned that when the typing issue settles down in future Python releases, the syntax may change enough that they would have to be redone.

I'm not worried. A few searches/replaces would likely do the trick.

Edward

[David Szent-Györgyi]

Edward writes:

I'm not worried. A few searches/replaces would likely do the trick.

That's good to hear. That tells me that you are sure that you can follow as mypy adapts to the changes that follow PEP 649. 

What interests me is that the Python development community has not settled on a solution to the problems posed by forward references. Those developers are so far from stupid that my first assumption must be that there's more to this than appears at the outsider's first glance. Such a first glance led me to suggest simply separating text annotations from the mechanisms described in PEP 649, but I am not an implementor of Python! 

[Edward K. Ream]

On Sun, Jun 27, 2021 at 5:14 PM David Szent-Györgyi <das...@gmail.com> wrote:

Edward writes:

I'm not worried. A few searches/replaces would likely do the trick.

That's good to hear. That tells me that you are sure that you can follow as mypy adapts to the changes that follow PEP 649. 

I have no intention of wading through the details of 649.

Right now, mypy is surprisingly intuitive and easy to use. I just finished fully annotating leoAst.py, and I can say that using strings to handle forward references is fine.

There are many possible future pitfalls that we could discuss, but I'd rather not :-) Let's leave future problems for the future.

Edward