ENB: No need to translitere tokenize.py into Nim

20 views
Skip to first unread message

Edward K. Ream

unread,
Jan 13, 2024, 5:21:21 PMJan 13
to leo-editor

Speeding up Leo's beautifier involves transliterating leoTokens.py into Nim. I had also planned to transliterate Python's tokenize module into Nim. 


But today I discovered that tokenize.py is a thin wrapper around C code! Transliterating and testing this C code would be a nightmare.


This discovery caused me a few minutes of consternation. My agony turned to ecstasy when I discovered nimpy, the Nim-Python bridge.


Summary


Three lines of Nim code should give Leo's fast beautifier access to Python's fast tokenizer module.

There is no need to transliterate (and test!) a Nim version of tokenize.py!!


Edward

Reply all
Reply to author
Forward
0 new messages