PR #3870 merged into devel

[Edward K. Ream]

The devel branch now contains PR #3870. This PR concludes all planned work on Leo's beautifier, leoTokens.py.

Lazy evaluation

leoTokens.py now generates whitespace using lazy evaluation. It writes whitespace only when the beautifier is sure that the whitespace is correct.

Previously, the beautifier wrote possibly redundant whitespace to a list of output tokens. A peephole optimizer then removed redundant tokens.

Imo, lazy evaluation is conceptually more elegant. But at the code level, lazy evaluation is (disappointingly) just as complex as a peephole optimizer. Furthermore, Lazy evaluation yields no significant increase in performance.

Ironically, the only speedup came from changing Leo's beautifier scripts. The beautify_leo.py script now beautifies only changed files and only looks at files in the leo/core and leo/commands directories. Only the beautify_all_leo.py script beautifies all files. I run this script before merging branches.

Summary

The new approach was worth doing, but just barely. It improved neither the performance nor the simplicity of the code. The only significant performance boost came from improved beautifier scripts.

Edward