ENB: The check_leo script *won't* use static analysis

[Edward K. Ream]

This Engineering Notebook post explains why (Aha!) static analysis can't possibly improve the check_leo script (PR #4484).

The existing script

The script creates live objects corresponding to Leo's fundamental objects: c, g, and p, as well as Python strings.

The script contains only a single ast visitor: visit_Attribute. This visitor scans attribute chains whose base (first attribute) is c, g, p, or s, including variants such as c1, p1, s1... In other words, visit_Attribute understands Leo's naming conventions. This visitor contains multiple (mostly Leo-specific) hacks that allow as much analysis as possible.

Aha: live objects are the ground truth

Yesterday, I spent about an hour in the tub thinking about how I might rewrite the script using static analysis. This analysis must include fully and accurately resolving imports, resolving the scope of classes, methods, and functions, and computing the contents of all namespaces. This would be a huge job.

This morning, I saw that Python's dynamic features make such an effort futile! For example, Leo's decorators and plugins can (and do) inject names and objects into Leo's commander class. Static analysis will (Doh!) never be as accurate as Leo's actual objects. It's as simple as that. End of story.

Improving the script

The present script has several limitations:

- It does not scan attribute chains whose base is the name of a class.

- It does not understand Leo's naming conventions involving w, widget, wrapper, etc.

- It stops scanning attribute chains containing function calls, array indexes, or dictionary lookups.

Once I stopped obsessing about static analysis, I saw that straightforward hacks can remove these limitations!

- Maintain context by adding visitors for ast.Module, ast.ClassDef, and ast.FunctionDef.

- Use that context to resolve attribute chains whose bases are arguments to functions and methods.

- Create live objects for more classes.

- Support more of Leo's naming conventions.

- etc. :-)

A model for other application checkers

The existing script contains two beautifully simple gems: the main line of visit_Attribute, and a crucial helper, visitor.split_Attribute. I'm proud of them—they can't be improved. The gems form the "bones" of the script. Others might use these gems to create their own application-specific scripts.

Conclusions

A Leo-specific script is the only way to find Leo-specific problems. Yesterday, I described the script to Rebecca as a microscope for examining Leo's code. The script contains dozens of special cases contained (elegantly enough) in various lists and dicts. I'll soon add more hacks and special cases.

Devs for other projects might use the check_leo script (especially the gems) to create their own application-specific scripts.

The script found bugs that even Félix's extremely careful review missed. To be fair, the script also highlighted two bugs in leoserver.py that Félix did flag but did not fix.

mypy and pylint do sophisticated analysis that is beyond my comprehension. But even those immensely complex programs have no chance of finding the mistakes that check_leo has just found.

This post marks a personal milestone. I have no further interest in understanding, much less improving, programs such as mypy, pylint, or pyflakes. I feel liberated. Onward to other adventures!

Edward