Rewriting of Python to Javascript translation

[Pierre Quentel]

Hello,

I have just committed a major rewriting of the Python to JS translation engine.

In script py2js.js, there was a signle function called tokenize() that parsed the source code, identified the tokens (strings, names, operators etc.) and built the syntax tree. It was probably the most obscure part of the whole Brython code...

The new version delegates the parsing to a new script, python_tokenizer.js. It provides a Javascript generator that yields tokens with the same format as those produced by the CPython module tokenize.

In py2js.js, tokenize() is replaced by a function dispatch_tokens() that, as its name says, dispatches the tokens and builds the syntax tree.

The change makes the code hopefully easier to understand and to maintain. But, although I spent time testing the new version, it is so different from the previous one that it may have introduced regressions here and there...

[Ray Luo]

Thanks for the heads-up. I'll try to use the latest Brython from github in my development for testing.

Does Pierre's workflow include running test (semi)automation on each commit? I do see a "www/tests" directory in Brython's code base.

[Pierre Quentel]

Le dimanche 6 juin 2021 à 18:58:22 UTC+2, raylu...@gmail.com a écrit :

Thanks for the heads-up. I'll try to use the latest Brython from github in my development for testing.

Does Pierre's workflow include running test (semi)automation on each commit? I do see a "www/tests" directory in Brython's code base.

Before pushing to the Github repo, I always run the test suite at /tests/index.html. For this specific commit I also ran all the examples in the gallery and the demo page.

[Andy Lewis]

Hi Pierre

I have now tried running most of my packages (brywidgets, brySVG, brycharts, tessellate, tessmaker, polyominoes, tangrams) with the latest version of Brython, and apart from the issue I submitted earlier (which had no connection with your recent rewrite), everything is working as expected.

I just thought the more testing is done, the more confidence you can have that all your changes are working well.

Best wishes

Andy

[Pierre Quentel]

Le dimanche 6 juin 2021 à 22:07:38 UTC+2, andy3...@gmail.com a écrit :

Hi Pierre

I have now tried running most of my packages (brywidgets, brySVG, brycharts, tessellate, tessmaker, polyominoes, tangrams) with the latest version of Brython, and apart from the issue I submitted earlier (which had no connection with your recent rewrite), everything is working as expected.

I just thought the more testing is done, the more confidence you can have that all your changes are working well.

Best wishes

Andy

Thank you Andy, that's good news !