ENB: About leoTokens.py

[Edward K. Ream]

This Engineering Notebook post introduces a new core module: leoTokens.py. This file contains Leo's new token-oriented beautifier, discussed in this ENB. I plan to transliterate leoTokens.py into Nim. 

Background

PR #3757 contains the present code. It is still a work in progress. PR #3757 is complex! Three earlier PRs contributed to the work. I could not have refactored the code without Leo.

leoAst.py contains Leo's legacy beautifier. This beautifier uses the TokenOrderGenerator (TOG) class, also defined in leoAst.py.

Goals for leoTokens.py

- Fixed type annotations of all data, functions, and methods.

- Minimal code size.

- Minimal dependence on other Leo modules.

Aha: keep leoAst.py

My initial thought was to replace the legacy beautifier with the tokens-only beautifier. But this was a bad idea for at least three reasons:

- The Orange class (the legacy beautifier) is an excellent demo of how to use the TOG.

- We want to be able to compare the old and new beautifiers.

- Adding another class to leoAst.py would complicate an already too-complex module.

leoTokens.py

leoTokens.py started life as a complex merger of the PRs mentioned above. Details omitted!

I then removed unnecessary classes, methods, and functions. En route, I simplified annotations. This pruning process ended today: leoTokens.py contains the minimum code required!

I saw that some helpers in leoAst.py were unnecessary. Again, details omitted.

Summary

Separating leoTokens.py from leoAst.py has simplified both files. Similar remarks apply to the corresponding unit test files.

leoTokens.py is now as small as possible. It is also fully annotated. This preliminary work will make transliterating leoTokens.py into Nim much easier.

leoTokens.py is still a work in progress. I must demonstrate that a tokens-only beautifier can infer context without the help of a parse tree.

I am confident that token scanning can do the job. The existing unit tests will be my guide!

 Edward

[Thomas Passin]

On Saturday, January 6, 2024 at 7:31:52 AM UTC-5 Edward K. Ream wrote:

I could not have refactored the code without Leo.

It would be interesting to read more about this. Other IDEs claim to support refactoring.  How did Leo make it possible in a way that others would not have?

[Edward K. Ream]

On Sat, Jan 6, 2024 at 6:44 AM Thomas Passin <tbp1...@gmail.com> wrote:

It would be interesting to read more about this. Other IDEs claim to support refactoring.  How did Leo make it possible in a way that others would not have?

Libraries such as Rope have a fixed set of refactorings. The work I did was a complex merger of three PR's.

I was continually looking at cloned nodes containing lists of subsidiary nodes that could either be revised or removed. The processes are quite different.

The Leonine way of refactoring does not depend on rigid tools or ideas.

Edward

[Edward K. Ream]

On Saturday, January 6, 2024 at 7:17:41 AM UTC-6 Edward K. Ream wrote:

The work I did was a complex merger of three PR's.

This was, in essence, a Leonine version of git's cherry picking.

Cherry picking is generally frowned upon, but in this case I saw no alternative.

The ability to focus using clones allowed me to see patterns that are larger than refactoring patterns.

Dozens of cffs helped.

In short, refactoring with cffs and clones is the bees knees.

Edward