Leo's new beautifier is too fast to speed up!

[Edward K. Ream]

I have just closed two items and marked them "Won'tDo":

- #3662: Rewrite Leo's beautifier in Nim.

- PR #3717: Transliterate Orange class into Rust.

- Experiments show that it would be difficult/impossible to make Leo's new beautifier 10x faster! As a result, neither project is worth the effort.

- I do not wish to burden future maintainers of Leo with Nim or Rust code.

I have enjoyed learning and using Nim. It's quirky but brilliant :-)

Edward

[Edward K. Ream]

On Friday, February 16, 2024 at 4:33:16 AM UTC-6 Edward K. Ream wrote:

> Experiments show that it would be difficult/impossible to make Leo's new beautifier 10x faster!

Here are the timing statistics to beautify leoAst.py with Leo's new beautifier:

tbo: 0.1 sec. dirty: 0 checked: 1 beautified: 0 in leo\core\leoAst.py

0.1 seconds is 100 milliseconds, so to be 10x faster, the total budget for the faster beautifier would be only 10 milliseconds!

However, a Nim-only prototype takes 3 milliseconds to approximate tokenizing leoAst.py. A full Nim-only tokenizer surely would take significantly longer.

Finally, the following nimpy code takes about 500(!!) milliseconds to tokenize leoAst.py.

let text_stream = io.StringIO(contents)
let readline = text_stream.readline
let tokens = tokenize_module.generate_tokens(readline)

Sure, I may have made a newbie nimpy mistake, but it appears highly unlikely that nimpy can make Python's tokenizer module available at a reasonable speed.

Summary

Leo's new beautifier is surprisingly fast, probably because Python's tokenizer runs at C speed.

As a result, it would be difficult/impossible (in any language) to get a 10x speedup.

Edward