Status report re the unification of the ast and token worlds

[Edward K. Ream]

I have just created #1440 to record this project. The fstrings branch contains the work. The status:

This project is all-consuming. Everything else is just going to have to wait.

Yesterday I finished round 1 of the required infrastructure, and preliminary tests passed. This was an important milestone.

Processing Leo's biggest file, leoGlobals.py, immediately failed. I had forgotten about 'comment' tokens. Fixing this should be straightforward. Only the eat method will change.

Last week's study of asttokens reminds me that recursive descent tree traversal is not really valid. The traversal must be "flattened" so that python's stack doesn't overflow. I'll discuss how to do this in another thread.

Summary

Imo, this project is worth any amount of work, both for me personally and the wider python community.

I expect major progress by Thanksgiving. I will be most unhappy if I can't finish this project by the time Rebecca and I leave for Florida in December.

Edward

[Edward K. Ream]

On Wednesday, November 13, 2019 at 4:26:02 AM UTC-6, Edward K. Ream wrote:

> I have just created #1440 to record this project.

The first comment now tells why the project is important, lists code highlights, and tells why the code will be extraordinarily easy to test.

I'll be updating this comment as needed.  It will be pre-writing for the eventual public announcement.

Edward

[Edward K. Ream]

On Wednesday, November 13, 2019 at 12:52:20 PM UTC-6, Edward K. Ream wrote:

> I have just created #1440 to record this project.

I continue to update the first comment. To summarize recent work further:

1. Infrastructure is complete: The first draft of all code is present. Simple tests pass. Others fail.

Ast.brief_dump is a crucial visualization. It has already revealed problems.

2. TOG.eat needs more work.

TOG.eat is the only difficult part of the entire TokenOrderGenerator class. Everything else is boilerplate.

It is not yet clear and obvious that eat is correct. Perhaps in a day or two...

Happily, eat is also an ever-present unit test. Errors quickly announce themselves.

In short, everything is going more quickly and smoothly than I expected.

Edward