Aha: type annotations need never be part of Leo's distro!

[Edward K. Ream]

#1944 suggests adding annotations for Leo's most important files.

At present, within the ekr-annotate branch, mypy passes all of Leo's core files without complaint, with the settings in leo-editor/.mypy.ini.

Alas, many of the type annotations are redundant. The effect is a substantial increase in code clutter and a decrease in readability.

Yesterday I considered writing a script that would strip off the annotations and put the annotations in stub files. But this morning I saw a simpler way...

Aha: why bother adding and deleting annotations? Just do the type checking work in a permanent, working branch.

So here's the plan:

- As discussed in #1058 (branch info), the ekr-annotate branch will never be merged into devel (or master).

- Formally, the milestone for #1944 will always be the next release.

- Any worries about future directions of type annotations will be confined to the ekr-annotate branch.

Summary

We can get the benefits of static type checking without adding type annotations to Leo's official code base.

The ekr-annotate branch will be a permanent, working branch that will never be merged into devel. This branch will likely never be part of Leo's official distro. 

#2018 will merge all the bugs found by mypy into devel, without all the annotations.

All comments and suggestions are welcome.

Edward

[Edward K. Ream]

On Tuesday, June 29, 2021 at 5:01:18 AM UTC-5 Edward K. Ream wrote:

> As discussed in #1058 (branch info), the ekr-annotate branch will never be merged into devel (or master).

A little thought shows that this is a terrible plan :-) It would be unbearable to have a permanent branch whose code must continually be brought up to date with devel. Aside from sort-lived branches, I never want to have two versions of the same code if I can help it.

Still, the goal of not polluting Leo's official sources with largely redundant annotations seems reasonable.

My present plan is to create two new commands, informally known as wax-on and wax-off :-) The insert-annotation command will insert standard annotations into a file (signatures only), where Leo settings will define standard annotations for particular files, or groups of files. The delete-annotations command will remove only standard annotations, leaving all other annotations.

Hmm, I suppose these scripts will have to deal with return annotations somehow. I'll have to think about this.

That's all for now. The goal will be to fully complete the work on type annotations for Leo 6.4. The scripts might help run mypy later on, but we'll have to see. I'll say more about Leo and type checking in another thread.

Edward

[Edward K. Ream]

On Tuesday, June 29, 2021 at 5:01:18 AM UTC-5 Edward K. Ream wrote:

#1944 suggests adding annotations for Leo's most important files.

At present, within the ekr-annotate branch, mypy passes all of Leo's core files without complaint, with the settings in leo-editor/.mypy.ini.

Alas, many of the type annotations are redundant. The effect is a substantial increase in code clutter and a decrease in readability.

Let me summarize my thoughts about mypy, type annotations, and related topics.

Learning mypy

As I said in another thread, it took only a few days to become comfortable with mypy. mypy is a superb tool.

Annotations have revealed bugs

For example, ec.centerLine, in leo/commands/editCommands.py, mypy showed that the "cast" to int is necessary in:

n = (fillColumn - len(line)) / 2
ws = ' ' * int(n)  # mypy.

mypy isn't worth the cost of annotations

Imo, despite finding a few new bugs, the costs (in readability) of annotations outweigh the benefits of using mypy.

Instead, pyflakes, pylint, and especially coverage tests provide more protection without odious annotations.

Leo is a static program written in a dynamic language

The type annotations in the ekr-annotate branch confirm something that I have long suspected, namely that I do know the types of almost all the data that Leo uses :-)

Leo could be rewritten in C or Rust without compromising what Leo can do. Yes, python is a privileged language within Leo, but c.general_script_helper shows that python isn't that privileged :-)

Nevertheless, there are at least two huge advantages to using python:

1. python handles storage allocation automatically. There is no need to handle storage allocation manually (as in C) or keep track of the lifetimes of objects, as in Rust.

2. The python interpreter provides a safe and fast environment in which to run buggy programs.

Finally, slots show that Leo's plugins can customize Leo without altering Leo's Position and VNode classes.

The way forward

It would be feasible to merge the bug fixes from the ekr-annotate branch into devel by hand. I could then just delete the ekr-annotate branch.

However, a wax-off script would help remove the annotations, thereby mostly automating the merge. A few regex search/replace operations would constitute a prototype wax-off, but a proper Leo script seems like a better strategy.

The wax-off script would benefit from Leonine settings, in one or more @data nodes. These settings would tell wax-off which annotations merely state Leo's standard naming conventions. The wax-off script might remove such standard annotations, leaving behind more interesting annotations.

We would likely want several @data nodes because which annotations are considered standard might depend on the particular file. The standard annotations for leoAst.py will be very different from the standard annotations for the rest of Leo.

Similarly, a wax-on script would insert standard annotations, using @data nodes to discover which annotations are standard.

Once the wax-on and wax-off scripts work, I'll convert them to Leo commands, say insert-annotations and delete-annotations. These commands will add or remove annotations on the selected file or files, guided by @data nodes. These commands might be of general interest.

Summary

Using mypy has taught me the value and limits of type checking.  My long-time interest in static type checking is nearing its end.  For me, type annotations are not worth their visual cost.

The insert-annotations and delete-annotations will add or remove annotations on the selected file or files, guided by @data nodes. I'll use the delete-annotations command to remove some (or all!) type annotations from a new ekr-annotate branch, thereby allowing me to merge a few bug fixes into devel semi-automatically.

The remaining work on annotations should take about a week. The work could be pushed back to Leo 6.5 if necessary, but there is no way that the ekr-annotate branch will become a permanent branch.

Your comments, please.

Edward

[Edward K. Ream]

On Tuesday, June 29, 2021 at 12:12:40 PM UTC-5 Edward K. Ream wrote:

> I'll use the delete-annotations command to remove some (or all!) type annotations from a new ekr-annotate branch.

BTW, leoNodes.py in devel already contains annotations. I'll probably remove those annotations as well.

Edward

[Edward K. Ream]

On Tuesday, June 29, 2021 at 12:12:40 PM UTC-5 Edward K. Ream wrote:

> My long-time interest in static type checking is nearing its end.

One more thing. The make_stub_files projects seems to be completely pointless now. I am going to update the readme to discourage its use.

Edward

[tbp1...@gmail.com]

We could use a standardized docstring format, similar to what I often use;

def func1(msg, reps):

    """Returns a sequence of random permutations of a message.

       ARGUMENTS

       msg:string -- the message to be permuted.

       reps:int -- the number of permutations to generate.

       RETURNS

       list:string   # or whatever the right annotation syntax is.

    """

This should be easy to parse and generate an annotated version to run mypy against.  There would be no extra code clutter, and many would say that a docstring like this should always be written anyway.  A docstring parser like this might even be a good addition to mypy.

[Edward K. Ream]

On Tue, Jun 29, 2021 at 4:41 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

We could use a standardized docstring format

An inventive idea.  However, that adds even more verbiage.  I think I know a better way. I'll write up my idea later today.

Edward