ENB: No need to translitere tokenize.py into Nim

Skip to first unread message

Edward K. Ream

Jan 13, 2024, 5:21:21 PMJan 13
to leo-editor

Speeding up Leo's beautifier involves transliterating leoTokens.py into Nim. I had also planned to transliterate Python's tokenize module into Nim. 

But today I discovered that tokenize.py is a thin wrapper around C code! Transliterating and testing this C code would be a nightmare.

This discovery caused me a few minutes of consternation. My agony turned to ecstasy when I discovered nimpy, the Nim-Python bridge.


Three lines of Nim code should give Leo's fast beautifier access to Python's fast tokenizer module.

There is no need to transliterate (and test!) a Nim version of tokenize.py!!


Reply all
Reply to author
0 new messages