Accessing the Skulpt tokenizer

[Tim Martin]

Hi,

I'm interested in tokenizing Python code from Javascript without compiling it to AST. I figured I could use the Skulpt tokenizer for that. But the tokenizer seems to be private; AIUI in older releases it was available as Sk.Tokenizer but got moved to Sk._tokenize. I guess I can access the latter, but it looks like it's not intended as a stable public API.

Would it be reasonable to turn the tokenizer into a stable and documented public API? The other alternative is to implement the Python tokenize library as a wrapper around the Skulpt tokenizer - that way the only API that needs to be stable is the Python one. It's a bit more of a faff to use for my use case, but it would be perfectly acceptable.

I'm happy to write a patch for either solution, but I wanted to get a sense of which solution is likely to be more popular.

Tim