ENB: Second progress report for leoTokens.py

[Edward K. Ream]

Progress has been unexpectedly slow, but the results are way better than expected! This ENB discusses the highlights.

Milestones

Creating a separate file gave me a clean slate to play with. The new code diverges in many subtle ways from the old.

I finally got a firm grip on performance issues. The code generators contained a rare O(N**2) botch. The code now gathers statistics that reveal such problems.

The fix for the botch revealed new design principles:

- The parser looks ahead (at most) one input token.

- The code generators look behind (at most) one output token.

- The parser creates all context for the code generators.

- The code generators create no additional context.

These principles guarantee O(N) performance.

To do

The parser must scan expressions to determine the exact context of slices (arrays) and dictionaries. This task will require managing additional semantic data. It's no big deal :-)

Summary

A new script file, tbo.cmd, contains four ways of calling the new beautifier. One of these ways highlights performance statistics.

The parser simplifies the code generators and guarantees lightning-fast execution. The new beautifier is twice as fast as the old!

The final result will seem easy and obvious. A look at the 250+ commits of PR #3757 should dispel this notion :-)

Edward