The mypy chronicles

[Edward K. Ream]

This thread starts with a copy of the discussions page for mypy issue #12352.

I am moving the thread here for the following reasons:

- To ensure the discussion will remain after my clone of mypy disappears.

- To reach the Leo audience.

- To demonstrate how I go about learning a complex program.

  There may be lessons for future Leo devs.

Edward

[Edward K. Ream]

This post summarizes what I have learned about mypy. I had planned to include the mypy the discussions here, but I won't do so. The details would detract from the summaries I am about to give.

In the early 1980's I enjoyed playing the Colossal Cave Adventure game. Well, the process of learning mypy is like playing that game, but it's a lot more fun :-) There is no right way to explore. Indeed, "mistakes" (and surprises) burn concepts into memory.

Here is an (approximate) chronological list of my mistakes and Ahas:

Mistake 1: Modify only TypeChecker.check_for_missing_annotations. Wrong! Adding default annotations to kwarg arguments should allow all checks to continue to work unchanged. Otoh, many unit tests may fail because they no longer give errors!

Mistake 2: Use g.SherlockTracer to trace the dynamic calls. I spent a day improving Sherlock. Alas, the traces gave too many details and too few insights.

Aha 1: Node.__str__ and Type.__repr__ are best for both users and devs. These methods pretty-print ast and type nodes by walking their respective trees with specialized classes. At first I thought the serialize methods would give me better data, but __str__ and __repr__ are perfect!

Aha 2: All *.accept methods are tree visitors. In traces, a "visit" method (including another accept!) always follows accept.

The next two Ahas relate to transportation, the ability to find necessary data.

Aha 3: in TA.visit_callable_type, t.definition is either None or a FuncDef. The PR might not be possible without the link back to the FuncDef. The "t" var does not provide enough data.

Aha 4: The SemanticAnalyzer (SA) class is more than just a tree walker! The SA class manages the entire semantic analysis phase. This class contains many useful ivars. For example, sa.modules [sa.cur_mod_id] contains the MypyFile (wraps an ast tree) for the current file.

Finally, these two Ahas relate to the big picture:

Aha5: load_graph (in build.py) parses all relevant files. It builds a graph for the builtins module that drags in many surprising files. At present, I don't understand why all these files are necessary. One would think that mypy needs special-case code for only Python's built-in functions. This Aha explains why the SA calls sa.analyze_func_def so often. Happily, Aha 4 allows us to filter out extraneous calls.

Aha 6: At some point (I don't remember when), I realized that only the SA class would need to change. There's no need to understand type inference!

Summary

This project is not quite retirement or a vacation. Nevertheless, it's great fun:

1. I give myself permission to make many false starts and mistakes.

2. There is no time pressure. I have stated that there is no timetable for the PR.

3. Exploring code with all of Leo's tools is inherently fun.

My primary research tools are traces in three methods. I have spent days refining the output as I learn what data are most important. These semi-permanent traces show patterns that pdb can't.

Edward

[Edward K. Ream]

On Fri, Jun 10, 2022 at 11:27 AM Edward K. Ream <edre...@gmail.com> wrote:

This post summarizes what I have learned about mypy. I had planned to include the mypy the discussions here, but I won't do so. The details would detract from the summaries I am about to give.

A few more notes:

Writing is an important part of the journey. The Ahas aren't earth-shaking, but writing helps clarify and consolidate my thinking. In other words, I'm mostly writing for myself :-)

Also, I'm convinced that the struggle itself cements learning. For me, reading doesn't have nearly the staying power.

Edward

[Edward K. Ream]

On Friday, June 10, 2022 at 1:16:11 PM UTC-5 Edward K. Ream wrote:

A few more notes:

And one more. leo-editor-contrib contains a recent version of mypy.leo. This outline contains notes, cff's, etc that reflect my journey.

Edward

[Edward K. Ream]

On Friday, June 10, 2022 at 11:27:04 AM UTC-5 Edward K. Ream wrote:

This post summarizes what I have learned about mypy.

Here's the most recent Aha: Use the assignment logic to "model" the new code.

The code is question is SA.process_type_annotation. It's a helper for SA.visit_assignment_stmt.

Edward