Unifying the token and ast worlds: status report

[Edward K. Ream]

This is #1440. On Christmas day the Fstringify class passed all its unit tests.  Yesterday I polished the code and ran additional tests.

Imo, this class is now good enough to be used as the basis of Leo's fstringify command. However, considerable packaging work remains.

Testing

Yesterday I fstringified leoAtFile.py from the "master" branch. This version of the file contains fewer f-strings, so is a better test. There were several valid warnings about strings followed by an "%" that could not be fstringified.  For example:

'%s %s' % delims # delims is a 2-tuple.

Such strings must be converted by hand, or just left alone.

Packaging

leoAst.py contains all the work. It imports only standard python library functions at the top level. Anyone should be able to use leoAst.py without installing Leo.

The LeoGlobals class contains simplified versions of several functions in leoGlobals.py.  Two functions contain:

import leo.core.leoGlobals as leo_g.

Only unit tests in unitTest.leo call these two functions, so importing leo.core.leoGlobals in them is harmless.

Remaining work

1. Add "traditional" unit tests to leoAst.py.  Anyone should be able to run these tests without installing Leo.

2. Convert Leo's own fstringify command so it uses the new classes.

3. Several experimental improvements. I'll discuss them if and when they work.

Summary

This project has succeeded. Anyone should be able to use leoAst.py now, without installing Leo.

I'll soon convert existing unit tests in unitTest.leo to traditional unit tests, probably contained within leoAst.py.

Considerable work remains.

Edward