What if we redesigned Brython ?

[Denis Migdal]

Hi,

I was bored on a train without Internet, so I started to think.

This are just the though of a bored mind, around a big what if.

This is purely theoretical, I just found the though exercice funny.

As you know Brython is more than 10 y.o., lot of code has been written since its beginning, nearly 9,000 commits in the current repository.

But what if, one day, we redesigned it ? What would could we do ?

Of course, we could use more modern tools, like ES6 classes, TypeScript, do some refactor, improve some tools, use web tools like webpack, etc. but what about the architecture ?

I think the primary focus would be on stability and maintainability. Indeed, we can see that Brython regularly have lot of git issues. Having a CPython-compliant Brython is more important than having the fastest Brython, after all, in Web GUI, we don't click a button 1,000 times a second, and as long as the computations finishes in 1/60th of a second, we're generally good.

Split code into 3 levels:

I think, a first thing would be to split Brython code into 3 levels:
- level 1 : A python subset, the core that knows almost nothing, only "if", "while", "functions" (only with positional arguments, no functions meta-data), floats, strings, low level imports, no integers, no PEP errors, no traces, no JS2PY nor PY2JS conversions.

This part is the most critical and should be the most stable.

- level 2 : core modules, they add features to the python subset in order to reach full CPython compliance. For example an "async" module would be defined as a directory :
/core-modules
     |-- /async
          |---- AST.js/transpilation.js/AST2JS.js/Py2AST.js/whatever.js (code to plug the AST/transpilation in order to handle async keyword in Py to JS transpilation).

          |---- runtime.js (code to handle the async features during runtime)
          |---- conversions.js (code to handle py2js and js2py conversions to add)
          |---- ... can have other files to split big files if needed.

To enable a module, simply create a symlink in a directory :
/code-modules-enabled
    |-- /async -> ../core-modules/async

Why using a module system ?

I think this simplifies greatly :
- code readability and maintenance.
- test/debug/beta-testing operation.

Modules should be almost independent enabling to add/remove/substitute them, which could be quite practical for development. This forces them to be quite independent, to follow the same structure, which generally reduces the number of bugs and facilitate code readability and maintenance.
1) We could automatically grade the stability of each module depending on the changes made to its files during e.g. the last 6 months. This would give a great overview of Brython stability status.
2) A module could also have a development cycle :
- dev : we are working on a new version.
- unstable : enabled in a Brython unstable version, to allow users to test it for some weeks/months
- stable/current
- legacy : not maintained anymore, we can  keep it if we still need it, e.g. to validate results, as a fallback, etc. e.g. for the function argument parsing, we can keep the unoptimized one just in case.

Having unstable version could enable to prevent issues with users wanting to use the latest version of Brython in production, that have lot of issues when new version is pushed. Waiting few weeks to produce a stable version would enable to fix the biggest bugs found by users.

Of course, Brython would only guarantee that modules works well with all other modules, and will not care about all the possible (and endless) combinations. By default all current/stable modules would be enabled.

In conclusion this module architecture is more for development/test purposes and for some very expert users that knows what they do at their own risks.

Note: we could assume that everything, even level 1 features, are implemented as level 2 core modules, and just mark them as being more important.

- level 3 : libraries, they are the least critical.

JS code integration

I think it could be great to have a way to easily integrate JS code into Python files at 2 levels :
- level 1 : internal, manipulating raw Py objets (not converted),  useful internally instead of splitting code into .py and .js in different folders.
- level 2 : external, manipulating converted Py object (i.e. after py2jsobj).

For example, list could have methods implemented as :
class List:
    def whatever(self):
           self.append(3)

@@begin_internal_js@@
__BRYTHON__.addMethod(__BRYTHON__.List,
                                                 "args1, args2",
                                                 function append({arg1, arg2}) {
                                                        // ...
                                                        // arguments and return values are raw Py objects, no conversions.
                                                 });
@@end_internal_js@@

Split Py to AST, AST to JS, and runtime steps :

This could enable to not having to load the code required for the transpilation if no transpilation is needed (e.g. AoT transpilation, JS code already transpiled and cached, etc.).

We could have a tool to visualize the AST graph for educational purposes, for documentations purposes, etc.


So this was the though of a bored man.
Dunno if you found this message diverting or boring, but feel free to share your thoughts, it could give me ideas to torment some students next year.

I think I will try to put some students projets on it for fun.
Last year I didn't had lot of students in my Brython projects, but I can try next year.

[Denis Migdal]

I started to play around a little.

I used Brython AST parser to get an AST tree I can then work on (I am really not familiar with Brython's parsing).
I rewrote the AST tree to have a more uniform structure so that I can print it (left : Python code, middle : AST tree, right JS code) :

I think I will have pretty soon some small proof of concept, that I will publish on github.
@Pierre : is this okay if I name it "SimplierBrython" ? I really have no inspiration for the name xD.

My goals aren't the same as Brython :
- I want the transpiler to be as simple as possible for educational purposes (I want to put student projects on it, so that they can play with it).
- I want the generated JS to be as simple as possible to read for educational purposes (I want students to be able to read it).

- I will not implement all Python features, only some basic ones.

- I don't care to be fully CPython compliant.
- I don't care to be able to support Python libraries.

- I don't care about performances.
- I only care about stability.
- I will publish some PoC, for AST tree structure, AST tree printing, unit tests, documentations, etc. that I hope could be interesting and potentially adapted for Brython.

- I think some design choices I'll make could be interesting for Brython, so it could serve as a PoC on the way some features can be implemented/architectured.

Just think of it as a playground for educational purposes and some PoC for Brython.

@Pierre: is it okay if I talk about this project here ?

[Pierre Quentel]

Salut David,

Thanks for the proposal. I understand your point of view : working on the translation of a Python-like language to Javascript is an exciting challenge for students as it makes them work on language grammars, parsing, ASTs, code generation etc. Using parts of the existing Brython implementation would simplify the task instead of rewriting everything from scratch. I would like to follow the development of such a project, and if you want to put Brython in its name it's ok for me (choosing a name could be a first challenge for your students !)

Rewriting Brython with radically different choices (Typescript instead of vanilla Javascript for instance) is such a huge task ! And there are still so many things to do - I have been working on a compliant XML package for more than 6 months now... So I leave this to developers with more computer science knowledge than I (I am mostly self-taught) and more up to date with the new technologies (I don't use IDEs and autocompletion, I hardly know anything about the typing system, I am lost with CI tools, etc.)

Cordialement,
Pierre

[mint...@gmail.com]

Sorry if I'm bringing too much overview thinking into this discussion.
I think it's very important, from time to time, to step back and look at things from a higher perspective.

It might be worth taking a few steps further and thinking about the whole area of ​​web applications.

What is the current use of web applications and how will their use evolve?

Are web browsers a good tool for meeting the needs of web applications, or do their inappropriate designs make many things more complicated and less efficient? Is there an appropriate technology on the horizon that would be better suited to running web applications than current browsers?

Is the transpilation of hundreds of languages ​​into JS is still the right way to integrate other programming languages ​​on the client side and then ensure their compatibility?

Is the concept used by Brython still beneficial compared to others solutions, especially say WebAssembly based technologies?

And there are many more, fundamental questions that would be wise to think about before you throw your efforts into the daunting task of redesigning Brython.

I do not invite a long-winded discussion, but I bring food for thought. I hope that this kind of reflection can be helpful.

Dne neděle 14. července 2024 v 9:11:16 UTC+2 uživatel pierre....@gmail.com napsal:

[Denis Migdal]

> Is the concept used by Brython still beneficial compared to others solutions, especially say WebAssembly based technologies?

This is indeed possible to convert Python interpreter into WebAssembly, and has lot of merits, like being 100% CPython compliant, being quite fast to convert, etc.

However, it also has drawbacks:
- Python interpreter is often quite heavy, not really great for non-heavy web applications.
- WebAssembly is executed in WebWorkers, so interacting with the DOM is a little troublesome.

> Is there an appropriate technology on the horizon that would be better suited to running web applications than current browsers?

There are indeed some frameworks that uses some lightweight browser to have some "desktop web app". The drawback is that this requires installing it like any other applications.

> And there are many more, fundamental questions that would be wise to think about before you throw your efforts into the daunting task of redesigning Brython.

Well, as I do not intend to be PEP compliant nor full CPython compliant, it becomes waaay easier.
After, the work will be done mostly by students for educational purposes.

[Denis Migdal]

Some more design choices :

1) No conversions: Use Py/JS types as is in JS/Py. I will try to implement things so that Python objects can be directly used in JS code, and vice versa.

Indeed, I saw that, in Brython, objects conversions are complex, and prone to errors/bugs.

I think Brython could be conversion-free, but it would requires to change how types are represented internally.

Some things may be challenging, but I think they can be solved :
- dict => JS object implicit conversion for JS functions arguments : I think there is a way to implement it, so that Python dict can be directly used as JS objects.
- functions call/argument parsing : I will only use positional-only arguments.
But I think python functions could also be implemented as a foo({arg1,arg2}) or foo(arg1,arg2) or foo(arg1, {arg2}) JS functions, with the arguments parsing performed outside of the function body.
- async functions : I will start executing it directly upon calls, and return a JS Promise.
To be PEP compliant, an alternate implementation would be to implement it as a JS async function and :
     - when called in Python : not call it, but generate a coroutine.

     - when called in JS : it starts executing and returns a promise.
     - when a coroutine is awaited in JS : do nothing, you have to convert it yourself to a JS Promise (or a method can be given for the JS API, e.g. coroutine.await() ).
 
2) Enforce Python type annotations :

I think I can vastly simplifies the generated JS if the variables types are known at transcription time.
So I think I will try to see if I can get the python type annotations from Brython's AST, and force their indication.

Also, I could have some student projects to work on an option that would disable such behavior, thus producing some more complex JS code but without requiring type annotations.

3) Option for PEP/CPython compliancy : Generate more complex JS code to be more PEP/CPython compliant. I won't do it, it'll likely be some students projects.

This would enable the following options :
- debug : true, more explicit error message, more checks, etc.
- async as coroutine : true
- enforce type annotations : false

[Denis Migdal]

I think I'm done with the core architecture.

Currently, I use Brython to validate the execution results:

As you can see, we can be quite fast if we take some freedoms (60x quicker and code ~100x smaller than Brython).
So if one day you have python codes that only add integers, you know what to use xD.

Now, I only need to:
- write the git repository's README.md
- write an unit test system: very quick to do now that I have the editor.
- add ~15+ python "features" that should cover most needs.

I will update once all of this is done, then I'll stop my work here.
This is quite fun to do, so I'll implement some "easy" features enabling to code some stuff, but I don't want to work too much on it.


Notably, I do not think I will add class declarations (I though of using JS mixins to implement them) for now as it is a big chunk of work.
I'll let students do the remaining work next year ^^.

[Joao S. O. Bueno]

...

> 2) Enforce Python type annotations :

Hi - just a diagonal reading here -
but please note that if at any point you do this (enforcing annotations),
your resulting language is not Python anymore.
Not by a far shot.

On Sun, Jul 14, 2024 at 4:49 PM Denis Migdal <denis....@gmail.com> wrote:
>
> I think I'm done with the core architecture.
>
> Currently, I use Brython to validate the execution results:
>

> --
> You received this message because you are subscribed to the Google Groups "brython" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/brython/5729f1d4-7e24-44c1-9199-a5a91d8efd6cn%40googlegroups.com.

[Denis Migdal]

A new day of work, new "features" implemented:
We can now use int, bool, None types, assignations, type hint annotations, if with bool condition, == comparison, and call functions with positional arguments.

Brython is helping me a lot for:
- parsing the Python code (ofc as I use Brython's AST).
- validating the generated JS code.
- avoiding some pitfall thanks to Brython experience.

I currently use a KISS/TDD approach: how to implement the most common use cases in the most simple way ?
It helps keeping a code simple and readable. In this regard, and as expected, type hint annotations help me a lot.

Of course, it is not enough for the remaining uses cases. For that I plan to generate the JS code differently for these use cases, while keeping it simple for the "common use cases". For example, I can write (a+b) in JS if I know a and b types at transcription, otherwise, I could do as Brython: _b_.__add__(a,b).

For the stack trace, I think I can convert the JS stacktrace into Python stacktrace thanks to my AST tree (that I could convert into a kind of sourcemap).
This would enable me to not have the positions arrays [0,0,5] everywhere in the code generated JS code.

I recently used this approach (TDD+KISS) on another project, and I think it has some merits:
- development is quick for the most used/important features.
- code is often cleaner, more readable, with less bug.
- the more the generated JS code is "pretty", the more easier it is to debug.

Being fully PEP compliant would require lot of work, and the more I advance in this project, the more I fully grasp how much Pierre have courage maintaining Brython.I do not think it will be difficult, but more quite time consuming to thoroughly go through the Python specs.

And I'm not even talking about all the Python libraries rewritten in JS xD.

For the benchmarks, it seems Brython poor execution speed last time came from the way I imported browser.console. Now I have similar performances.

On my side AST convertion is taking a little time, but it doesn't really matter:
- transcription can be AoT if performances are an issue.

- and in the worst case, Python parsing could be rewritten to speed up this process.

One interesting thing that could be done, is using SBrython to benchmark Brython runtime execution speed.
Indeed, if we want to compare Brython vs vanilla JS, it requires to rewrite by hand the Python code into JS.
With SBrython, we can automatically generate JS that is quite close to what we would have in vanilla JS.

For the generated JS script, I'm, as expected, smaller than Brython, and I think I'll have a runtime library quite smaller (I kind of cheat by not being fully PEP compliant xD). One interesting thing, is that I think it could be quite possible and easy to do some tree shaking with SBrython, to build a runtime library containing only what is strictly necessary for execution. This is something I can try out when I'll have something in it.

[Denis Migdal]

> Hi - just a diagonal reading here -
> but please note that if at any point you do this (enforcing annotations),
> your resulting language is not Python anymore.
> Not by a far shot.

Thanks for your read ;)

TBH, I just started by what was the most easy/practical for me, in order to progress quickly.

It will be possible to disable it through options, or to customize its behavior.

This is manly for me, to help me when debugging, e.g. by having an optional runtime type checking after each expression (got the idea while answering your message ^^). Currently, this also generate an AST with indications of the resulting type for each expression.

Indeed, as I do not yet support all operations on all types, having a way to check the types helps me quite a lot.

Currently, this is the behavior I have implemented:
- Values without an annotation type hint have either their type deduced, or a type of "unknown".
- I only enforce variables' type hint or deduced type.

I think I will do the following changes to help myself:
- in the editor, running the code twice : one with and one without runtime checks.
- forbid unknown types except for JS function call return value.

Such behavior could be very easily changed though options, to be more or less strict e.g. :
- ignore variables type annotations.
- forbid (or raise a warning) explicit type violation .
- forbid (or raise a warning) on variable deduced type violation.

- forbid (or raise a warning) all non-explicit unknown type.

- forbid (or raise a warning) when using values of unknown type.
- runtime type checking.

Personally, I kind of like stricter checks as it helps detecting bugs quite quickly/easily for a very small price, while auto-documenting code.
After, I came from JS... which is literally a living hell without type checking. And before that I did C/C++ where we focus more on performance than being user-proof. So I guess I do not quite have the Python mindset ^^.

I also find it quite useful for educational purpose:
- it enables to generate clean JS code.
- it quickly screams on students when they make mistakes ^^.


After, maybe my way of doing will hit a wall, but that's what PoC are for ^^

[Denis Migdal]

As there are some architecture/design choices that requires experience, I am currently writing a first version.

Then students would work on writing/improving tools, documentation, AST generation, or features.

I already made an editor (that only supports integer and additions xD), and still haven't made the runtime.

However, I think this can already interest you: you can hover part of the code to highlight it in the Python code/AST/JavaScript code:

See live example at: https://denis-migdal.github.io/SimplerBrython/

I think this can be potentially adapted for Brython and could be quite useful when debugging (not always easy to find the JS code corresponding to a Python line).

This only requires AST nodes to contain (can be ofc modified/adapted in Brython) :

{

    pycode: { // to be able to highlight the node in the Python code

        start: {line: 0, col: 0},

        end: {line: 0, col: 6}

   },
   jscode: { // to be able to highlight the node in the JavaScript code - personally I add it when I generate the JS code.

        start: {line: 0, col: 0},

        end: {line: 0, col: 12}

   },

   children: [] // to be able to highlight the node's children in orange.

   type/value // used to print the AST node.

}

See the documentation : https://denis-migdal.github.io/SimplerBrython/doc/Transpilation.html

Ofc the code is still quite dirty, and surely have 1 or 2 bugs.

--

You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/brython/c7a69547-13e2-4eb9-aab8-1cbf47862e84n%40googlegroups.com.

[Denis Migdal]

I am now able to find the AST node corresponding to a JS stacktrace line.

I will open an issue on Brython's github as it could simplify stack printing (?) without having to simulate it by hand.
Indeed, in the past, Brython had some issues related to stack printing, often due to JS errors not properly handled by Brython.
I will explain more the technical details/limitation on the github issue.

Currently I use it to, when hovering a line on the output, highlight of the corresponding JS/Python/AST:

I also now support putting "core modules" into subdirectories (you just need to create a directory, and they are automagically imported), I thing this greatly make the code more readable/sorted:

@Joao : some good news, I though about your message once more, and decided to perform my type checks AFTER the AST is generated, and before code execution. Meaning that it would be very easy to skip such steps, or to add/remove some checks. I think I will implement it today. This could also enable to highlight parts of the code on which there is a warning/type check failed.

Though I do not plan to support operations on unknown type yet, lot of more important features to implement first.

[Denis Migdal]

Loops and functions (pos only args) now implemented.
Currently I seem to be 2-4x faster than Brython, with a code 7.5x smaller.

I still have to implement:
- Exceptions
- list/tuple/dicts
- import/export

So this is where the fun begins (cf below) ^^.

I'll also have to implement some operators, and keywords, but they aren't really a brain twisting challenge.
Ofc, being fully PEP-compliant would take waaaaay more work. My type deduction/checks would also require some more work.

Exceptions:

For the exceptions, the issue is that I can catch both Error thrown in JS, and Exception raised in Python. And I still do not want conversions. So I decided that Errors and Exceptions are NOT the same things, and to process them as follow:

- raise:
- `Exception` => converted to `throw new PythonError(Exception)` (`PythonError` extends `Error`).
- `Error`     => forbidden.
- Except:
- `PythonError` (i.e. raised in Python) extract the stored `Exception` python class.
- `Error`       (i.e. thrown in JS)     wrap it into the python `JSException` class (inheriting `Exception`).

Errors and Exceptions are not the same thing, so these are not conversions, therefore it's okay ^^.

Though, I dunno how rethrow would behave in such implementation...
I'll also need to convert the whole stacktrace from JS to Python (and to make it compatible with Chrome/Chromium as they have a different stacktrace format).
I also won't support catching non-Error exception (in JS you can throw strings, but that's really really dirty).

List/Tuple/Dict:

I want to implement them using JS objects:
- list : JS Array
- tuple: Object.freeze(Array)
- dict : JS objects.

For that I will try to design a system of symbol substitutions (if the type is known at transcription time):
- e.g. list.append(1) would be written as list.push(1).

For classes, in a more general way, I want to try the opposite of what Brython is currently doing:
Currently Brython is doing most of the transcription (or conversion) when declaring the Python class (or converting a JS obj into Py obj), and then manipulating a "Python object". Instead, I want to manipulate a "JavaScript object" and adapt the operations.

For example :
`foo + 2` would be written in JS as `_b_.add(foo, 2)`.
With `add` implemented e.g. as `a.__add__(a, b)`.

The issue is that `__add__` isn't defined in existing JS classes.
But I can add a symbol in their prototype and write something like:
```
function getProto(a) {
     return a[$BRY_PROTO] ?? a;
}
function add(a,b) {
     return getProto(a).__add__(a,b)
}
```

Meaning that I could use existing JS classes with a completely different prototype as long as a $BRY_PROTO as been defined on their prototype.

In an analogous way, if I need to, I can also store Python-specific data in classes instances using a symbol.
Could be used e.g. for non-string keys in dict.

Also, I'd like calls on a class (e.g. `Klass()`) to be converted as `new Klass()`.
We can recognize some classes by looking at their toString(), and I could cache some info in their proto.

Some other classes seem to be implemented as functions, but it seems simply calling them as functions works... so I guess that'd be okay ?


Now, I'll take some time to think about it to not go too fast.

[Denis Migdal]

Exceptions/try/catch added. Time for more design dicussions about argument parsing and null/undefined/None.

None:

I first considered that null and undefined were both None:
- implicit None is undefined (e.g. function returning without specifying None).
- explicit None is null.

The advantage is that, when calling JS functions from Python, we'll get None as the return value (undefined).
The main drawback is that null/undefined are different in JS, so we might have some strange behavior, s.a. storing 2 None (null and undefined) when using JS Set and Map. But why not.

But then I noticed that:
```javascript
foo(a=2){}
foo(null)      //a=null
foo(undefined) //a=2
```
When in Python:
```python
foo(a=2):
foo(None)      // a=None
```

This would therefore generate some inconsistencies. It is therefore better to (as Brython is currently doing) assume that only null is None (both implicit and explicit), and that undefined is a different special value, with undefined == None and undefined is not None.

Some JS functions would therefore return undefined instead of None. Though, if the function returns nothing, we'd often call it without even collecting its return value. And when undefined is explicitly returned, well, it'd be a special value.

Therefore, if the last line of a python function isn't a return, then I'll add a `return null` as the last line of the generated JS function.

I'll also need to implement my first import for this.

Argument parsing

I started to think on how to represent a python function parameters in JS.
Contrary to Brython, I don't want to implement it as `function foo(...args)` requiring to parse the argument inside the function. I want to parse the arguments outside the function body, i.e. when calling it, enabling to perform quicker calls when called internally (or from JS code), e.g.:

```javascript
function foo(???) {}

function call(fct, args) {
fct( ...fct.parseArgs(args) );
}

call(foo, args);
```

I could therefore make it `function foo(locals)` or `function foo({a,b,c})`, but that wouldn't be practical/nice when calling it from JS.

I decided to implement it as : `function foo(...pos_only, ...pos_args_without_default_values, {...kwonlyargs, ...pos_args_with_default_values, kwargs: {} = {}} = {}, ...vargs)`, e.g.:

```python
def foo(a, b=2, c=3):
```
```javascript
function foo(a, {b=2, c=3} = {});
```

This has several merits :
- quite nice to call from JS.
- pretty sure that, in lot of cases, I could pre-parse parameters at transcription time instead of runtime.
- I could set the default values inside the function signature (making calls in JS nicer).

This as some drawbacks for PEP compliance:
- order of arguments aren't guaranteed (...vargs being at the end)
- function signature can be modified in Python to some extends.

In both cases, forcing the call to the parsing function, would solve the issue. Although, signature changes wouldn't be reflected when called from JS... but I do not consider it to be an issue.

Also, when explicitly giving undefined to a parameter, it'll be set to its default value... but I think this behavior is okay as undefined would be considered as a special value.

Using JS function in Python

For existing JS functions, I plan to use `toString()` to assert its parameters enabling to use fully use it in Python e.g. with keyword arguments.

For [native code] JS functions, I'll provide a way to patch such functions define its signature.

[Denis Migdal]

Hi,

No student took my Brython projects, so today I tried to create a PoC for a very small (and very incomplete) py2ast parser for fun.

For a Python script of 200 lines, the time to generate JS went from 25ms (Brython) to 1ms (SBrython).

Please be reminded that I do not care to be fully Python-compliant so I ignore lot of checks that Python/Brython are doing.

Still, Brython py2js process seems to be quite slow.

I wonder if it could be possible to speed up the py2js transcription by:

- distributing the scripts to convert across a pool of webworkers (potentially a x8 speed up ?).

- re-implementing the py2js process into a langage able to manage allocations (e.g. C++) then converting it into WASM ?

After, TBH, if we really want performances AoT transcription would be way easier to manage, so Brython py2js speed isn't that of an issue.

For now, I won't use my new parser for now as it is very incomplete, and I have other priorities (SBrython is missing a lot of core features).

Even though there are some incompatibilities between Brython and SBrython, maybe it would be possible for some project to use Brython in the dev environment (for the standard Python checks/errors messages), then try to use SBrython in the production environment if more performances are needed (ofc when SBrython will have more features).

I'll take my afternoon to implement some missing core features.

Maybe in few weeks I'll try to run some Brython student project with SBrython, or to run some Brython unit tests with SBrython.

Cordially,

[Ed]

Hi Denis, that's interesting.  I missed the original emails and just had a read.

I see value in this project as some of your improvements might find their way into Brython proper (where possible considering the very different design goals).  It's always good to rethink and explore alternatives.

What are your thoughts are on how SBrython compares to other python-like transpiler projects, such as Transcrypt, Skulpt, and PScript?  I've tried all of them and found them lacking for my purposes.  Mainly because they change Python semantics via unsupported syntax, behavioral differences, js quirks, and other incompatibilities.

I don't require 100% python compatibility, especially in libs.  But often I find the differences to be unpredictable and obscure.  There's just too many hidden gotchas that bite me when I code with them.  Not worth the frustration.  Still, I do occasionally use them in rare cases where Brython isn't suitable.

If SBrython is closer to a pure python subset minus a few features (async can go to hell, I avoid it even in cpython),  that would be interesting to me.

Would also be interested if SBrython supports precompiling python scripts to js.  Runtime compilation doesn't work where dynamic code execution is forbidden, such as chrome extensions with manifest 3 (required after 2024).  Brython can conceivably do this, but so far it hasn't been implemented for understandable reasons (Pierre has to prioritize and it's not a common need).

Thanks for the update.

Ed

To view this discussion on the web visit https://groups.google.com/d/msgid/brython/dbfb389b-5b54-43ef-9031-62627a6c0a76n%40googlegroups.com.

[Denis Migdal]

Hi,

Thanks for your message.

> Brython can conceivably do this, but so far it hasn't been implemented for understandable reasons (Pierre has to prioritize and it's not a common need).

Use this: https://github.com/denis-migdal/Brython-AOT

I can fix it if there is issues.

> What are your thoughts are on how SBrython compares to other python-like transpiler projects, such as Transcrypt, Skulpt, and PScript?

I don't remember them all, but IIRC lot of them can't directly access DOM and JS from the Python code.
Brython was the best transpiler I found when I searched for one, the most Python-compliant, up-to-date, lightweight and reactive.

I'll try to make SBrython interoperable with Brython (cf next mail) enabling devs to use SBrython for performances, and Brython for the wide standard library support.

Skulpt is a Python2 interpreter, so kind of obsolete.
PScript hasn't been updated for 3 years and the generated code not that nice.

Transcrypt is very heavy IIRC.

SBrython aims to generate clean (and fast) JS code, without requiring JS<->Py objects conversions that are a pain in the ass as shown by the list of Brython issues due to it. It should be easier to debug from the JS side, and easier to interact with JS APIs.

SBrython aims at remaining simple. In its base form, lot of Python edges cases/features will be ignored, SBrython will only support a subset of Python.

SBrython has a modular architecture enabling to very easily change the way the JS code is generated from the AST. In this regard, I plan to have a "compatibility" mode, with several options, where the generated code would be python-compliant at the cost of speed and code simplicity.

I wanted to have some students to work on it, but didn't got any on my projects :'(

For now, I'll try to pass parts Brython unit tests suites then I think I'll work on Brython <-> SBrython interrop.

> Would also be interested if SBrython supports precompiling python scripts to js.

It does. But currently SBrython lacks LOT of features (I'll work on it during this WE).

Cordially,

[Denis Migdal]

@Pierre : I wonder if, theoretically, SBrython could help Brython for porting Python modules to Brython (ofc once I finish to pass parts of the Brython unit tests suite).

Indeed, currently Brython requires to rewrite C/C++ Python modules into JS.
Maybe it could be possible to use SBrython to write Brython core code (i.e. the "internal JS") in Python without losing performances ? It would likely requires some helpers functions, but it could be possible.

The advantage is that people could help porting C/C++ modules without having to know JS, and may be more confortable with Python compared to JS ?

Ofc, currently SBrython is VERY unstable, incomplete, and not recommended for use.
Once I finish to pass parts of the Brython unit tests suite, maybe I'll try to work on some Brython<->SBrython interactions.


Maybe, for performances, some users might also transpile Python modules written in Python with SBrython (with potentially personalized ast2js modules) then register them as a Brython module. It would be quite unsafe, but could be used when performances really matter.

[Denis Migdal]

Good news I passed the numbers and basic test suite... bad news I excluded 1393 tests out of 2112 xD.

Currently, Brython has ~8k lines of tests, if we only take the core language features.
Some unit test files are quite long (~1k lines) so I try to split them using the existing comments, in order to more easily exclude tests on non-implemented features.

One issue I had is that tests are kind of "out of order", e.g. tests on complex numbers are dispersed in the unit test file instead of being gathered at the same place.

I have an exclude list file with some keywords in order to more easily find the relevant tests when I implement a new features... well the exclude list file currently has 400+ lines xD.

Currently I replaced the asserts so that it doesn't throw, enabling me to test the whole file.

Now I'll try to add, little by little, new tests to pass.

[Ed]

Brython can conceivably do this, but so far it hasn't been implemented for understandable reasons (Pierre has to prioritize and it's not a common need).

Use this: https://github.com/denis-migdal/Brython-AOT

Thanks, I'll give it a try.  

[Denis Migdal]

I will now works on the operators/base type, but before that it seems I'll have to work on the types representation.

I'll use some static type infos for static analysis for the code optimization upon generation. A structure like that:
type = {
    attr: {substitutions, final, types[]} // e.g. list.__len__() could be remplaced by list.length in JS.
}

I though a limitation would be I'll have to forbid classes prototype manipulations, but Python has "@dataclass(frozen=True)".

Meaning that I'll simply have to require classes to be frozen in order to this optimization to be applied (with an option to make all classes frozen by default).

Another constraint is that I have to know the variable type at transcription type.

However, most of the time it is known.

The only issue is if the variable is re-affected with a different type inside a flow control struct (if/loop/etc).
In such case the optimizations can't be made once we are out of the if or at the start of the loop.

I think that Brython could make some optimizations of this kind for some variables/classes. I dunno if the effort is worth it though.

I also improved the unit test system :

https://denis-migdal.github.io/SimplerBrython/tools/Editor/index.html?test=brython
(disable privacy.reduceTimerPrecision on FF for better precision)

Currently, SBrython reduces the generated code by 85%.
69 lines passed/1922 (1853 lines excluded)
In total I have ~149 sub-tests that are excluded. Once operated are implemented, I should be able to un-exclude <61 more + <51 (for some functions) + <19 (for exceptions) + <13 (for __op__) sub-tests

[dgront]

Dear all,

it's a. very interesting discussion. I'm looking forward future developments. As for me,

- re-implementing the py2js process into a langage able to manage allocations (e.g. C++) then converting it into WASM ?

I ve been programming in rust for a few years, I would recommend that as a language to write a WASM translator. I've also spend two decades or so in C++, so have some perspective. Rust offers much better support in building WASM modules.

One should keep in mind though that loading a WASM module has also its overheads.

Best,

Dominik Gront

 

[Denis Migdal]

Binary operators **, /, %, //, *, +, -  implemented.
69/1917 tests passed.
SBrython is ~2x faster during runtime, even though I use BigInt for integers (normally I should be much slower as Brython uses numbers).

Though, I have a little strategy to optimize int to float conversions by doing them during transcription when possible.

For example 1/1 => Number(1n)/Number(1n) => 1./1.

This could be improved, e.g. by detecting that an int can be replaced by a float without impacting runtime behavior.
But that'd be a pain to implement:
1) build some kind of graph to see which variables the int influences... with the issue that the graph might have cycles.
2) detect where a conversion would change behavior (requiring to reconvert it before use).

3) Compare the number of conversions to float with the number of conversion to int, to see if the conversion during transcription is worth it (and what would then be the ratio ?)... but would be hard if there are loops...

A more greedy approach would be to first put all int literals as float, and to convert then into int on the first use requiring and int.

But another issue is that float doesn't have the infinite precision of int. So we could have a loss of precision if there are some "overflow"...

Finally, the best solution is to wait for browsers to optimize the implementation of bigint xD

Py2AST is quite slow compared to my AST2JS. Maybe I should work on it one day.

For the operators I use a structure to represent their behavior :
__add__: {

    return_type(type) => type == 'int' ? 'int' : NOT_IMPLEMENTED, // int + int => int

    call_substitute(a, b) => r`${a}+${b}` // a.__add__(b) is written a + b

}

I used some helpers to help me generating theses structures to avoid repetitions (and still have a lot):

    ...GenBinaryOperator('sub', { // declare __sub__ and __rsub__
        return_type: {'float': 'float', 'int': 'float'}, // float - int -> float / int - float -> float / float - float -> float
        convert: (a) => a.result_type === 'int' ? Int2Float(a) : a,
        call_substitute: (node: ASTNode, a: ASTNode, b: ASTNode) => {// __sub(a, b)__ = __rsub(b,a)__ = a-b
            return binary_jsop(node, a, '-', b); // binary_jsop can add parenthesis to ensure operator precedence
        }
    }),

I still have lot of operators to implement (mainly comparison operators).
Then I guess the next step would be handling complex function argument parsing (for now I only use pos only args) in order to then implement lists and tuples.
So see you next week I suppose ;)

[Denis Migdal]

Passed 108/1907 tests (1799 excluded), -85% size, -44% exec time.

Implemented lot of operators for int/float/str/None/bool.
Still have some to implements (bytes, compare int, compare float, all assignation operators, type(), isinstance(), type constructors and __int__() operators).

Some quick design considerations below (and some longer considerations after):
- I could precompute some comparisons when I know they'd be false due to the type. However, this might remove function calls with side-effects.
- I could precompute some operations on literals. However, I believe this will hurt JS code readability.
- for some bit-wise operator, I could transform 1n|1n to BigInt(1|1), should be faster, but would requires some graph operations on the AST.
- some substitutions I'm making would be illegal if we authorize inheriting base types and then redefining the operators (could be an option to toggle).
    -> if we know that the left operand is a literal, we can still make the substitutions.

And know some discussions about comparisons.
In Python we can chain comparisons, that's nice to use, a little less to implement xD.
Brython implements 1 < 2 < 3 as $B.rich_comp('__lt__', 1, locals.$op = 2) && $B.rich_comp('__lt__', locals.$op, locals.$op = 3).

One issue, the last locals.$op = 3 seems unnecessary and might, in VERY particular situation cause memory leaks. I'd advise to write :
$B.rich_comp('__lt__', 1, locals.$op = 2) && $B.rich_comp('__lt__', locals.$op, (locals.$op = null, 3) )

With substitution we get:
1 < (locals.$op = 2) && locals.$op < (locals.$op = null, 3).

One thing is that under some conditions, we can repeat the operands :
1 < 2 && 2 < 3.

This is possible only when :
1) the operand is a literal.
2) the operand is a single symbol (e.g. locals.a)
3) the operand is a simple computation without side effects using literals or/and symbols... But what is a simple computation ? 1 operation ? 2 ? 3 ?

This is the same issue with a += 1 which is a shorthand for a = a + 1.
Currently operators are substituted as `${a}${op}${...}`. I'll have to find a system to indicate/detect when `${a}${op}=${...}` is authorized.
The issue is that a //= 1 might be implemented as a = Math.floor(a/1) with the right operand being evaluated first.
So if we have foo().x //= 1, we have an issue, there doesn't seem to have a possible simple substitution. The only way is doing it the Brython way:
(obj= foo(), key='x', obj[key] = Math.floor(obj[key] + 1).

Coming back to comparaison operators.
Ofc, we'd like to write something like $B.compare(1, '<', 2, '<' 3). But this won't work as all operands will be evaluated, when 1 < 2 && 2 < 3, 3 isn't evaluated if 1 < 2 is false.

and/or behaves the same, but in JS, && and || behaves similarly to Python so no big difficulties on this side.

Also, if 1.__lt__(a) is not defined, but a.__gt__(1) is, 1 < a would be substituted as a > 1, changing the operation order. I added a little tweak to fix that.


I have quite some fun playing with Python operators definitions.
I though I'd be finish sooner, but still have some work to continue on operators/base types before going to functions argument parsing.

[Denis Migdal]

Still lot of things to implement, but here some considerations about JS-Python types equivalences.

Python           JS
-------------------------------

str                   string
int                    bigint
float                number
bool                 boolean
list                   Array
tuple                Object.freeze(Array)
bytearray        Uint8Array() => requires cast to bigint/number when accessing/setting elements.

bytes               We can't freeze Uint8Array. The best would be to inherit Uint8Array or to add a [isFrozen] tag. Would be mutable on the JS side though...
MemoryView DataView ?
Set                  Set but with an issue. Python uses values (hash) when JS uses references...
                        JS Set could be viewed by Python as a JSSet extending Python Set (or having the same methods/operators). You could then either directly use it as a Set in Python, or to explicitly convert it as a Set to get the Python behavior.

Set[hash]        Python Set[py_obj with __hash__] could be implemented as a PySet extending JS Set and replacing its add/etc. methods so that it could be used on the JS side as a "special" set.

                         Inside PySet, a Set(hashes) OR a Map(hashes, values) ?

Set[no hash]   Python Set[int/float/jsobj/py_obj without __hash__?] could be internally replaced by a JS Set for performances (while being seen a Python Set) ?
FrozenSet       ROSet extending Set and redefining methods to prevent changes.

Dict[str]            {} => simplifies interactions with JS functions ? [and ofc pure JS objects are dict[str]).

                         could use Object.create(null) to have a clean JS object, but as we'd accept JS objects as dict[str] we'd have to handle junk symbols any way (and Object.create(null) is slow).
                         JS objects could be seen as JSObj extending Dict[str, any] [?] enabling to explicitly converting it into a full Dict.
Dict[any]         Map, same issue with Set, and same solution.
                         => need an explicit cast before giving it as argument to a JS function expecting a JSObj.

The fact that JS Set()/Map()/{} are seen in Python as JSSet/JSMap/JSObj enable developers to use them like a Python Set/Dict or to explicitly cast them (at the cost of a copy) to ensure a pure Python behavior. Static type checking should prevent issues.

To get "true" python Set/Dict, simply using their constructor should be enough :
set( JSSet )                => copy and get a true Python set.
dict( JSMap/JSObj ) => copy and get a true Python dict.

The fact that Set[no hash], Dict[str], Dict[no hash] created in Python are seen in Python JSSet/JSMap/JSObj could be a little unsound, but I think necessary.
Maybe, instead of being seen as a subclass, we could have instead some helper functions :
import sb;
sb.isJSSet(a) / sb.isJSMap(a) / sb.isJSObj(a)
sb.toSet(a) / sb.toDict(a)  // <- return a if already a, else call set() / dict().


The more I think, the more I understand there is a LOT to implement xD.

[Pierre Quentel]

One issue, the last locals.$op = 3 seems unnecessary and might, in VERY particular situation cause memory leaks. I'd advise to write :
$B.rich_comp('__lt__', 1, locals.$op = 2) && $B.rich_comp('__lt__', locals.$op, (locals.$op = null, 3) )

Good point Denis, I have removed the last assignment in commit 7d914b0

[Denis Migdal]

Long story short, I fixed and proved the test system stats generation (cf below), so lets discuss performances a bit.


Warning: the results are for ~96 tests in 15 files, with lot of assert. So this is not representative of what you would get for a single file of 96 lines (due to some overcost).
Warning: please be reminded that SBrython aim to implement a subset of Python, and is doing static analysis to optimise the generated code.
SBrython is therefore faster than Brython, mainly because it doesn't do lot of checks Brython need to do.

Currently, on the pure runtime, I'm x12 faster than Brython.

Note: the tests are mainly on int operations (so quite slow for me as I use BigInt).

I can still improve it by :
- transforming some int into float when possible.

- precomputing some int values when possible (impact readability).

Comparatively, the browser takes a lot of time parsing the JS (x20 more than the pure execution time).
Though be reminded that in real life, you would have part of the code called multiple times (functions, loops).

In this regard, I am x1.2 faster than Brython, and x2 faster in the total execution time (JS parsing by the browser + execution).
The only way to improve performances is to reduce the size of the generated code (x6.5 less generated code compared to Brython).

I could improve it by:
- removing JS indentation and new lines.
- pre-computing some int

There are other ways to improve it with:
- a pool of shared WebWorkers to load, then parse many big files at once.
- tree shaking.
- remove dead code.
- name mangling.

But in this case, you'd better to AoT transcription and use existing JS tools (minifier, uglyfier, bundles, etc).

For JIT, we have to convert the Python code to an AST, then to JS.
This is ~10x slower than the total runtime (and ~100x slower than the pure runtime), I'm currently x1.15 faster than Brython.

TBF I should also take into account the time spent by the browser for parsing the JS code of the python parser.

My JS file is currently 398kio (including the unit test system, contains the sourcemap) vs 1,1Mio for Brython.js, but I use Brython AST (not included in my JS file).

There are few ways to optimize it :
- generate SBrython.js with production options (remove sourcemap, minify, remove comments).
- if AoT, remove the py2js code and only keep the runtime library (currently my runtime library is very very small).
- if JIT on user-generated code, you could also do the py2js transcription on a server...
- disable features you don't use (just comment a line in one file and rebuild).

Currently we really can't compare the file sizes (and their loading time in the browser) as I still miss a LOT of features to implement.
At the end, I think it is highly likely that my SBrython.js will be as heavy as brython.js.

Let's go back to py2js.

Most of the time (>90%) is taken by the Python code parsing.

I think this is possible to make a parser that'd be x10 faster by dropping most of the checks, assuming we are creating a JS code for "production".
For Brython, it is ~66% taken by py2ast.

My AST conversion (for my own need) is a little slower than my AST2JS. But as the bottleneck is currently Py2AST, this is not an issue.
My AST2JS is x4.2 faster than Brython.

One thing is that Py2AST only need to be significantly slower than genFct which is an unsurpassable bottleneck (4.1ms).
If we assume my own parser would be x10 faster, I'd still want to be way faster (7ms vs 4.1ms), and in this case, lot of ASTConv code would disappear.

This could be improved by :
- using struct when implemented by browsers.
- generate the JS code without using template string literals functions ( r`${a}xxx` )... (but on the other side I want my code to stay readable).
- waiting 20 years for the PC to be 100x faster.

Status : SUCCESS

Tested : 96/1909 (1813 excluded) [15]

Code size : (x 6.53/-84.68%)

Executed in : 37.520ms (x 1.16/-13.43%)

Runtime : 4.380ms (x 1.94/-48.50%)

genFct : 4.120ms (x 1.20/-16.60%)

exeFct : 0.260ms (x11.77/-91.50%)

Py2JS : 33.400ms (x 1.15/-13.02%)

Py2AST : 28.340ms

ASTConv: 2.640ms

AST2JS : 2.420ms (x 4.16/-75.94%)

[Ed]

Performance metrics are rather interesting.  I've optimized cpython code and seen small changes make a big difference.  Once I changed `os.dirname (path)` to `path.rsplit ('/',1) [0]` for a noticeable improvement when looping on 1 million paths.

That said, I only optimize when 1) current code is noticeably slow and 2) profiling shows a bottleneck.  Day to day, I write the most obvious readable code and don't worry much about performance.  Like the old saying: premature optimization is the root of all evil.

Languages, compilers, interpreters, etc are different.  You never know what parts of your code will be used heavily, so you have to anticipate and optimize everything to some degree.  I applaud Denis's efforts to look at where things can be improved.  And Pierre's efforts at improving Brython during the time I've used it.  You guys have done remarkably well with a very tough job.

From my perspective, here's what I would like to see as an end user:

• the biggest improvement would be better load time.  I don't know whether that's dominated by parsing time, retrieval time, generation time, or what the bottlenecks are.   I just notice that large brython scripts take awhile to load.  I don't pull in many modules, just a few shallow ones without deep dependencies.
• run time seems to be fine.  once brython script is loaded, I don't notice delays.  most of my brython code is user-event driven anyway (key press, mouse click, etc), so execution speed isn't usually a factor.  a few parts of my code do process many items (10k to 100k objects) where improvements could be made, but it's generally fast enough.
• one way to address load time is pregenerating js code.  convert python to js once on server side instead of paying the penalty client side with every page load.  Denis's BrythonAOT project is a great step.  It'd be nice if brython proper could support this or something similar.

Hope that helps.  Keep up the great work.

--

You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/brython/9f9de7d2-8e2d-4da9-b735-e3fad427b2b8n%40googlegroups.com.

[Denis Migdal]

> The biggest improvement would be better load time. I don't know whether that's dominated by parsing time, retrieval time, generation time, or what the bottlenecks are. I just notice that large brython scripts take awhile to load. I don't pull in many modules, just a few shallow ones without deep dependencies.

/!\ By default, Brython modules are loaded synchronously, trying several paths where the module could be located.

You can try to see in the network tabs of the developer tools if fetching your assets, brython.js, etc. takes too much time. If your site can be accessed publicly, I can try to take a look.

If the issue is here, there are ways to improve performances:
- generate Brython modules (?) so that Brython knows where to fetch the file.
- preload ressources.


I think the 2 other bottleneck are likely:
- the size of brython.js file (1Mo of code).
- py2ast+ast2js.

> a few parts of my code do process many items (10k to 100k objects) where improvements could be made, but it's generally fast enough.

When you do heavy DOM operations, remove the container from the DOM, do operations on the container (all the operations you want to make), before reinserting the container into the DOM. It will greatly speed up performances as the browser won't have to do paint/reflow.

DOM access are generally quite slow.

> Denis's BrythonAOT project is a great step.  It'd be nice if brython proper could support this or something similar.

Thanks. Did it worked for your project ?
I can also do an online version of the converter if needed.


/!\ Brython generates JS files that are way bigger than the Py source file (more than x5 bigger ?).
Though assets should be compressed by the server, and generated Brython files has lot of repeated structures so in reality it should be a lot smaller on the network ?

[Denis Migdal]

Some design discussion about int...

As you know, I'm optimizing int (bigint) operations by internally implicitly converting them as number when possible.
After some intense thinking, I think I need to explicit it with different internal types:
- int    : default, normal python int, implemented as bigint.

- jsint : small int literals, integer returned/taken by JS functions, etc. implemented as number.

This distinction, in fact, exists in Python.Python does have a notion of index-sized integer (defined by sys.maxsize). jsint would then be an index-sized integer.
Some Python structures, e.g. bytes/bytearrays would manipulate jsint as well.

Brython also has internally a notion of long (bigint) and short int (number).

However, Brython implicitly cast number returned by JS functions into int if it has no decimals. I find it a little dangerous.

With my method however, you would be able to indicate that a JS function returns a jsint, and not a float. I think it'd be safer.

When a function return or takes as parameter an int or a jsint, I'll be able to perform implicit int <-> jsint casts.
This would also prevent some useless casts, e.g. number -> bigint -> number.

One drawback of my method, is that if you push an jsint into a List[any] it might then be seen as a float.

I think I may have to convert them into int when passing it to a variable of unknown type.

I believe only * operations (with ofc unary -) could be optimized if will cast the result it into a float, so we can take the precision loss.
Maybe I could try to make one special rule for it.

Operations on jsint might produce int (bigint) to prevent loss of precision.
I could optimize it by estimating the log2 of the maximum value, to be able to detect when an overflow is likely to occur.
Some type hint could be provided e.g. uint8 that would be converted into an unsigned jsint of 8 bits.

Well it feels like micro-optimization...

By default integer variables will be of type int (to avoid type issues when +=).
It could be optimized if the variable is detected, or written as Final, e.g. `ID: Final = 1` (though Brython doesn't seem support this typehint ?).

However, the variable may also be used in a way bigint cast are constantly required...

Short literals will be of type jsint.
Strangely enough, doing `BigInt(1)` is as quick as `1n`. This would enable to perform the `BingInt()` cast as late as possible.
And it'd be easier to implement.

[Ed]

Thanks for the suggestions.  Here's a call tree example from one of my heaviest brython scripts (5k lines - maybe 3.5 to 4k without blank lines and comments).

It's about 6 seconds from invoking brython () to script completion.  My script starts at main () which takes 3.78 seconds in chart.  No surprise there - my script does a lot, including rewriting page and ajax queries to local servers.

That leaves 2.2 secs between brython () being invoked and main () starting.  My guess is most of that is parsing time, but not sure.  It's definitely not retrieval time.  I checked Network times as you suggested.  Each module import takes 3ms to retrieve, according to chrome network tab.  Which sounds right.  Because I run brython in a chrome extension, chrome translates module urls to a local file on SSD.  So 30 ms total to retrieve 10 modules.

Overall time is ok, it's tolerable, no complaints.  But faster is always nicer - especially startup time.

Thanks for the tip on removing DOM containers.  I'll give it a try.

Still playing with BrythonAOT.  Works for small tests, haven't tried a larger script yet.

[Denis Migdal]

> It's about 6 seconds from invoking brython () to script completion.

Ouch...
And me finding 13ms is too slow xD.

I do not have enough information, but something is clearly wrong somewhere.

Also Brython normally store the generated JS code into the indexDB to avoid re-transpiling it each time.
Maybe you should check with Pierre to see if the caching system really work.

@Pierre : could it be possible to have a __BRYTHON__.printStats() that will print the total time for :
- brython.js startup (before starting to parse the first script).
- downloading py scripts
- building the AST
- converting the AST into JS
- creating the Function from the JS code.

> Still playing with BrythonAOT.  Works for small tests, haven't tried a larger script yet.

I'm glad it worked on the first try. ^^

[ben levitt]

Hi Ed,

I run a pretty large brython project too, and I had ~4sec start times.  I got big improvements by making sure I had brython's indexedDB cacheing enabled (on webworkers too), and even more by using `brython-cli make_modules` to build a brython_modules.js file, that I then include instead of brython_stdlib.js.  This avoids multiple round-trips, silent 404 errors, and associated retries at alternate locations to get all the modules individually.  It took my load times from ~4sec to about 1sec for a large project.

Ben

--
You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/brython/CAF1%3DpDmORtciEQ%3DQTMe7sv%2BU2dXv115Gt6nea7wv3%3DCcvenmSw%40mail.gmail.com.

[Ed]

Thanks Ben, I'll look into that.  This particular application doesn't use any stdlib modules, but other apps do.

To view this discussion visit https://groups.google.com/d/msgid/brython/CAHyr3Yg8srQsMBirSx0zgdfPoOb7aq3ZRnacsFaEUzgJm172Yg%40mail.gmail.com.

[ben levitt]

Hi Ed,

The speedup is that your custom modules are added to the brython_modules.js file, and then the brython runtime doesn't need to look for and download each of your modules separately.  This is what made the biggest difference in load time for me.

Ben

To view this discussion visit https://groups.google.com/d/msgid/brython/CAF1%3DpDmN1vmB34GC9__86E2p%3DXGdhKyrzvP6wcSb47kWsSorBw%40mail.gmail.com.

[Denis Migdal]

jsint + other operators implemented.
108/1902 tests passed.
Currently pur runtime x15 faster than Brython (total runtime x2 faster), ast2js x3 faster.

a+=1 and a-= 1 becomes ++a / --a
Number(-a) becomes -Number(a)
BigInt(-a) becomes -BigInt(a).

Number(a*b) becomes Number(a)*Number(b).
Should be quicker most of the time. Could be optimized further, but do not think this would worth it.

Next steps :
- type constructors and their associated operators int() => int.__init__() + __int__().

- rework functions (how to represent their signatures, deduce their return type, complex signature).

    - add async/await => will enable to load JS modules with JS import().
    - add yield
    - type guards...

- add some global functions for the unit tests: type(), isinstance(), etc

After that, I'll just have to work a little on :
- classes
- Brython interactions
- splitting the generated files, write the doc, etc.

I won't support variables of unknown type to keep the runtime code as small as possible.
I could work on a compat mode, but I'm afraid that'd be too much of a work.

I do not know if I'll try to finish my own AST parser. That'd require some work when you could simply use AoT transcription.

[Denis Migdal]

Complex function definition done.
It seems I am x10 faster than Brython on function declarations.
I still hasn't implemented complex function calls, but I may be > x100 faster than Brython.

As the argument parsing is made AoT, it quite speeds up the runtime. Ofc this requires the function to be known during transcription.

The goal was to produce JS function that could be easily used in JS :

def foo(a, *t)              => (a, ...t)

def foo(a, *t, **args) => (a, t, args)
def foo(a    , b=3)      => (a, b)
def foo(a=2, b=3)      => ({a = 2, b = 3})
def foo(a=2, b=3,/)    => ( a = 2, b = 3 )

def foo(*, a)                => ({a})

I won't guarantee order of position parameters evaluation when using keyword arguments.
I could very easily do so, but either I'd have to put all non-posonly parameters inside a dict {}, or to do runtime argument parsing.
As always, I can add a compatibility mode to change this behavior.

Deduction of function return type done.

I can't correctly process recursive functions and out-of-order declarations.
For that I'd need to visit the AST tree differently: instead of using a recursive function, I'd need to have a stack, and a kind of in_waiting ASTNode set.

I think I'll do it at the very end when I'll have more time.

I'd like also to respect the original indentation when converting lists (e.g. arguments, arrays, etc) from Python to JS (for now arguments are all on the same line).

[Denis Migdal]

Erratum, forget to remove some console.log.

For function definitions :
- total execution time x11 faster
- browser JS parsing x3 faster
- pure runtime x50 faster

[Denis Migdal]

Let's talk a little about libraries size.
Ofc, the smaller the better for downloading speed and the browser parsing of the JS code.
Web server usually compress data before sending them, leading to ~x6.4 less data transferred (bz/lzma/br).

brython.min.js is 1.1MB, and becomes ~180kB after compression.
@Pierre : could it be possible to have an estimation of the weight of the AST parser (with all its dependencies ?).
If you can give me the list of all the files required for the py2ast, I can also compute it.

I estimated the current SBrython size (without compression) :
- runtime        <   14kB (including TS types indications, not minified).
- transpilator < 189kB (not including py2AST, including runtime, not minified, with some dead code).

SBrython runtime start up should therefore be really fast.
As the generated JS code is as clean as possible, you should nearly get the native JS performances.

At then end I hope my transpilator would be <500-600kB (x2 smaller than Brython.js).
Let's say +200kB do add unimplemented features +200kB for the AST parser.
Ofc, adding more (optional) features would increase its size.

In a few weeks/months, I'll need some testers to check if the implemented features are enough for everyday use.
The goal isn't to find edge cases or gotcha, but to see if SBrython is usable and if it can interact with Brython.

For that, I'd need a collection of real life code I could test. I think I'll be able to get some student code for that.
But could be great if somebody had some code (or part of code) we could test.

[Pierre Quentel]

@Pierre : could it be possible to have an estimation of the weight of the AST parser (with all its dependencies ?).
If you can give me the list of all the files required for the py2ast, I can also compute it.

The scripts used to generate the AST from the source code are
py2js.js
py_tokens.js
python_tokenizer.js
python_parser_peg_version.js
pegen.js
gen_parse.js
action_helpers_generated_version.js
string_parser.js
number_parser.js
py_ast_classes.js
py_ast.js
symtable.js

Some functions in other scripts may be called depending on the source code,
for instance operations on literals use py_utils.js and possibly py_int.js or
py_float.js.

Once the AST is generated, the translation to Javascript is done in ast_to_js.js.

[Ed]

For that, I'd need a collection of real life code I could test. I think I'll be able to get some student code for that.

But could be great if somebody had some code (or part of code) we could test.

I would love to help out with this.  Especially some longer / more complex scripts just to push the limits and see what breaks.

Unfortunately most of my brython scripts are for internal environment.  Lots of ajax calls to custom servers on private networks.  Mainly I build network apps with many parts (backend servers, data stores, etc).  Brython is the front end, doesn't do much without remote data.

I'll see if it's feasible to isolate some of the brython code, or perhaps replace remote calls with dummy data.  I already started looking, it's not a quick process.

I also have a few small self-contained scripts that I can share.  Scripts to: monitor DOM for images and add full-size popups on hover, magnifying glass zoom for image details, things like that.  Pretty simple code but maybe it helps.

I'll throw what I have in a new github project and share link when it's ready.  How soon will you need it?

To view this discussion visit https://groups.google.com/d/msgid/brython/bb0a2f69-a4bb-491a-aa01-b7d5e6cacff2n%40googlegroups.com.

[Denis Migdal]

Complex function call implemented.
Next step is implementing most of python builtin types and functions.
A little regression, classes do not longer work: the PoC was quite naive:
- it wasn't using static type information.
- it hasn't a true "self" context.
- it wasn't using __call__

For complex function call, the argument parsing is not quite optimized, could try to find a better algorithm/precompute more things, but seems to be fast enough (py2ast is still the bottleneck).

I think I could optimize the JS generation which, in my case, produces LOT of arrays recursively (lot of arrays of arrays of arrays of arrays).

This is due to the fact that I need to keep track of the position in the generated JS code, in order to better print stack traces.

But it would requires some benchmark and has py2ast is the bottleneck, I rather not optimise it yet.

@Pierre Thanks.
From what I get (including comments that are excluded from brython.js which is 1,114kB):

- runtime lib : ~200kB ;

- ast_to_js    : ~140kB ;

- ast              : ~233.71kB+773kB (gen_parse).

The more I work on SBrython, the more I understand how Brython works.
Lot of things I do in SBrython are similar to the way Brython is doing it.
Still I try to not look to much at Brython to have a fresh approach.

I think SBrython has 3 main differences with Brython :
1. SBrython uses static analysis to perform work Brython is doing runtime to the transcription time.

2. SBrython doesn't have Python<->JS object conversion.

3. SBrython is skipping lot of checks/edge cases.

If (1)&(2) greatly decrease runtime and the generated files size, it doesn't decrease the sbrython.js size.

For the (3) has the others it decreases runtime and the generated file size, and slightly decreases the sbrython.js size.

But I do not think the gain on sbrython.js is significative.

Therefore I think that at the end my sbrython.js should be ~600kB.

My runtime will still be very small has substitutions are performed during transcription.
If we want to be more python-compliant, I can generate runtime classes from the static type informations. Which would make the runtime a little bigger (14kB -> ~200kB?).

You'd have a decrease of performances, but still should be faster than Brython has there are no (2), and for (1), I'd still have some type informations.

I'd say there are few ways to decrease a code size :

- use helper functions to remove repeating structures/code.

- simplifying implementation / algorithms.

- removing some checks (but unsafe).

And to address the elephant in the room... gen_parse...
This is big, way too big. I think the code is adapted for local compiled code :

- no download: the file size is irrelevant ;

- compilation: so no script parsing time.

I really think a small parser can go from 900kB to 200kB.
I think a big parser is adapted for AoT and debugging, but when in production we should either use AoT, or use a smaller/more naive parser that'd skip lot of checks.

I kind of also dislike Brython AST nodes: each type has its own structure.
It is easier to read, but I find it harder to use.

I prefer them to have the same structure :
- type (the AST node type)
- result_type (static type info)
- children (AST node contained by this one).
- value (some arbitrary information not contained by an AST node, e.g. value of constants)
- toJS (convert node to JS).

- pycode/jscode (information about positions in the Python/JS code).

I'm also not a fan of Brython file hierarchy.

I think some files in src/ should be split into different directories to facilitate browsing the project:

- types/ for py_int, py_float, py_types, etc.
- parser/
- transcription/ for ast2js, split into several files.
- helpers/

[Denis Migdal]

@Edwartt: Thanks.

There are nothing urgent.
I won't run out of work for quite a long time ^^.

Just do it when you can.
Depending on the feature your code use, I might need to implement things before using it for tests.

[Denis Migdal]

@Pierre: I may have a possible strategy to boost Brython startup time (cf below).

On performances again.
As I split the tests into separate executions (27 "files" for 167 tests), I wondered if it biais results... and it does.

I learned that:
1) It is significantly faster (up to 12x) to put all generated JS code into the same "file", then ask the browser to parse it once.
Though, I assume this over-cost is reduced on larger files (in my test I have only ~6 lines per file).
2) The AST parsing is very slow. In Brython:

- py2js is responsible (on the unit tests) of 86% of the total execution time (97% in SBrython).

- py2js is responsible for 97% of the startup time (99% in SBrython). So if you use AoT in Brython, you should have a 33x faster startup time.
     -> also if you use AoT, you could also use a smaller brython.js (2-4x smaller ?), so also a reduction on Brython startup time.
     -> @Pierre: could it be interesting to have the following strategy ?
           -> brython_runtime.js => fetch Python scripts and start executing scripts already transcripted to JS.
           -> brython_parser.js    => if a script hasn't been transcripted, brython_runtime.js fetch and load brython_parser.js to transcript it.
           -> detect top-level scripts and top-level imports and start fetching them asynchronously before the first execution and before parsing the file.
                  -> this would enable asynchronisme and even parallelization of the transcription process before the first execution.
                  -> this would also enable to test several URL at once to find the module asynchronously (before the first execution).
                  -> first loading of brython_parser.js could then be made asynchronous (before the first execution).
           -> could also use a pool of shared workers to transcript such fetched scripts before the first execution.
                  -> this may greatly improve startup time (e.g. x8 faster ?).
           -> promise.all() to wait for all top level scripts/imports to finish before running the first execution.
3) For SBrython, if we get a quicker py2ast, ast2js might become the new bottleneck (~5x slower than the JS code parsing by the browser).

I did plan a small refactor tomorrow (for other reasons) which may (or may not) slightly improve performances.
4) The JS code parsing by the browser isn't a bottleneck anymore (well, it still is for AoT startup time).

Indeed, by putting all the tests into the same "file" :

- JS parsing by the browser is 12x faster. And goes from being 1.2x faster than Brython to being 4.33x faster.
- overall runtime execution goes from being 2x faster than Brython to 19x faster.

- no significant changes to ASTConv/AST2JS.

- py2ast and pure runtime are slower due to the way I merged the tests like that:
def _1():
    # the test 1

_1()

[Pierre Quentel]

- py2js is responsible for 97% of the startup time (99% in SBrython). So if you use AoT in Brython, you should have a 33x faster startup time.
     -> also if you use AoT, you could also use a smaller brython.js (2-4x smaller ?), so also a reduction on Brython startup time

Since the beginning there have been countless suggestions to use AoT in Brython, and my position has not changed: by design, I want the development cycle to be as simple for Brython as for Javascript : edit the source code / reload the page. If you have an intermediate step (a program that translates Python to Javascript before running code in the browser), you have to run it every time you change the source code, and you also have to translate the whole code base (including the standard library) every time you change the translation engine

Not that if you include brython_stdlib.js in the HTML page, the translation of Python modules in the standard library is transparently stored in an indexedDB cache and only (transparently) changed when the Brython engine or brython_stdlib.js is updated; for average Brython users, this only happens at each new Brython release. You can observe the speed improvement if you add brython_stdlib.js to the integrated test suite at tests/index.html, the first run takes as long as without it, but the next ones are significantly faster. This is not exactly AoT, but greatly reduces the number of translations.

You can also group all  the Python scripts used by the application in brython_modules.js and let the Brython engine also transparently store the JS translation, but this requires using the CPython Brython package

[Denis Migdal]

Thanks for your answer.

A little question: does Brython uses the HTTP Last-Modified and If-Modified-Since HTTP headers to avoid re-downloading the scripts if not modified ?

Also what do you think about the strategy I suggested to speed up the firsts transcriptions ?
1) if no changes were made to the python scripts, and if you don't use compile()/exec()/etc you don't need to load the Brython parser into the Browser.
2) before the first execution, you load the firsts scripts asynchronously enabling to reduce waiting time on the first page visit.

This isn't AoT but could speed up initial page loading.

[Ed]

Thanks for sharing, Pierre.  Design decisions are very interesting.  It's good to understand your rationale.

In that light, I have a few questions:

Since the beginning there have been countless suggestions to use AoT in Brython, and my position has not changed: by design, I want the development cycle to be as simple for Brython as for Javascript : edit the source code / reload the page. If you have an intermediate step (a program that translates Python to Javascript before running code in the browser), you have to run it every time you change the source code, and you also have to translate the whole code base (including the standard library) every time you change the translation engine

1) What's the reason for wanting to make development cycle as simple as possible?  Is it because your primary audience is students / academic environment?

2) Is pre-compiled (transpiled / translated) code incompatible with simple development, ie no extra steps?  Isn't it possible to have both?  In fact I think brython does have both to some degree: brython runs straight python code, but  caches translation in browser as a speed optimization.

3) If speed optimizations are acceptable when they don't compromise developer simplicity, why only optimize the single user case?  Brython's browser caching optimization only speeds up loading (parsing / translation) for one user in one browser accessing the same page repeatedly.  Why not consider optimizations across the entire system, as long as developer simplicity remains unchanged?

4) What about optimizing speed for many users across many browsers?  Typical websites / web apps serve thousands or millions of users a day.  That duplicates a tremendous amount of work: every user's browser has to parse and translate the same static brython code on loading the page.  Why not have the server translate brython to js once per script, and serve the resulting js code to thousands or millions of users?  That saves time & energy across the system as a whole. If you run an internet site maybe you don't care much about duplicating end users' cpu time.  But if your brython app serves a large internal network then duplication matters.  It also speeds up first load for everyone if brython script is already parsed & compiled to js.

5) Isn't it possible to achieve precompilation as an optional step?  For instance: brython.js runs and finds scripts on page.  If it finds type "text/python" then it parses and compiles as currently happens.  If it finds a pre-compiled type, like say <script type="brython/javascript"/>, then brython knows to proceed without parsing / translating the code, just like if it found the code in indexedDB cache.  This gives developers the option to add better first-load performance and reduce resource duplication if needed.  While preserving the current simplicity for those who want it.

6) Couldn't precompilation be implemented invisibly, without even needing to explicitly run a compile cycle?  The web server can compile and cache brython scripts when it detects changes to the source, then serve the compiled result.  Any significant brython app needs to be served from more than just "python -m http.server".  In which case the server can be configured to handle precompilation transparently.  Developers and end users won't see any difference at all from how things work currently.  For cases where configuring a web server to do precompiles is too much effort, brython can continue to work exactly as it does now, parsing and translating python code on the fly in each browser.

Would you be willing to support more system-level optimizations as an optional step, as long as current development simplicity is maintained? 

[Denis Migdal]

> 2) Is pre-compiled (transpiled / translated) code incompatible with simple development, ie no extra steps?  Isn't it possible to have both?  In fact I think brython does have both to some degree: brython runs straight python code, but  caches translation in browser as a speed optimization.

There are no incompatibilities: I was able to dev Brython-AOT very easily.

I think the issue is that this would make more things to maintain while Pierre is maintaining alone Brython, a project that is already quite massive.

> Why not have the server translate brython to js once per script, and serve the resulting js code to thousands or millions of users?

Generally you'd have a build process to produce the JS file in your dev environment, and only push the JS files into the prod server.

IIRC Brython-AOT has a watch mode, enabling to re-generate the JS files when the Python source files are changed.

You can then use a standard webserver.

There are 3 other ways :

(1) having a fast-cgi to make the transcription (a little like what PHP is doing) ;

(2) having a server transcripting Python and serving the JS files ;

(3) having a transcription library that you could include into your own server.

If needed, I can do (3) with Brython AOT with a dummy example for (2).

> 5) Isn't it possible to achieve precompilation as an optional step?

Not needs for that, you can just insert the generated JS as a normal JS script.

> The web server can compile and cache brython scripts when it detects changes to the source, then serve the compiled result.

This is possible with ionotify functions.

However, keep in mind that Pierre is already doing lot and lot of works.

You can't expect him to do everything alone. If you want something, you can create it.

--
You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/brython/CAF1%3DpDnEfHmE_49LdrmwFEVpVTM%3DS4uQQjn2DwO61tbkhaqzVg%40mail.gmail.com.

[Ed]

I completely agree, Pierre has enough to do supporting Brython as is.  I'm not trying to give him more work.  Handling this type of optimization in a separate project with different devs is a good model.

But a separate project would benefit from basic support in regular Brython.  Such as making compiled outputs more accessible.  Brython already saves it to memory when running, and saves it to indexedDB for caching.  Adding a regular file output should be trivial, and makes it easier to run separate projects.

Brython-AOT is good, but not officially supported.  Creating & maintaining it requires quite a bit of knowledge of brython internals, it seems.  Few of us have Denis's proficiency at this (your updates about building a parser / transpiler make my head spin).  A regular js output would make things easier for the future.  With little extra cost to brython itself.

At least it appears that way to me.  Maybe I misunderstood something.  I know about web apps, not brython internals.

To view this discussion visit https://groups.google.com/d/msgid/brython/CAFr00W4z5ZEz1ja-gkPM_Eww8uG_YJkXaVWKmGo25aV%2BnTEE-Q%40mail.gmail.com.

[Pierre Quentel]

Le jeudi 31 octobre 2024 à 19:46:23 UTC+1, Edward Elliott a écrit :

Thanks for sharing, Pierre.  Design decisions are very interesting.  It's good to understand your rationale.

In that light, I have a few questions:

Since the beginning there have been countless suggestions to use AoT in Brython, and my position has not changed: by design, I want the development cycle to be as simple for Brython as for Javascript : edit the source code / reload the page. If you have an intermediate step (a program that translates Python to Javascript before running code in the browser), you have to run it every time you change the source code, and you also have to translate the whole code base (including the standard library) every time you change the translation engine

1) What's the reason for wanting to make development cycle as simple as possible?  Is it because your primary audience is students / academic environment?

No, it's because Brython is meant as an alternative to Javascript, so it must be as simple to use. With an AoT engine you would have to install a Python program, with all the potential issues with pip, compliance between the CPython version and the Brython version (compiling Brython 3.13 stdlib requires CPython 3.13). With the current situation, loading the Brython engine from a CDN is all you need

 

2) Is pre-compiled (transpiled / translated) code incompatible with simple development, ie no extra steps?  Isn't it possible to have both?  In fact I think brython does have both to some degree: brython runs straight python code, but  caches translation in browser as a speed optimization.

3) If speed optimizations are acceptable when they don't compromise developer simplicity, why only optimize the single user case?  Brython's browser caching optimization only speeds up loading (parsing / translation) for one user in one browser accessing the same page repeatedly.  Why not consider optimizations across the entire system, as long as developer simplicity remains unchanged?

4) What about optimizing speed for many users across many browsers?  Typical websites / web apps serve thousands or millions of users a day.  That duplicates a tremendous amount of work: every user's browser has to parse and translate the same static brython code on loading the page.  Why not have the server translate brython to js once per script, and serve the resulting js code to thousands or millions of users?  That saves time & energy across the system as a whole. If you run an internet site maybe you don't care much about duplicating end users' cpu time.  But if your brython app serves a large internal network then duplication matters.  It also speeds up first load for everyone if brython script is already parsed & compiled to js.

5) Isn't it possible to achieve precompilation as an optional step?  For instance: brython.js runs and finds scripts on page.  If it finds type "text/python" then it parses and compiles as currently happens.  If it finds a pre-compiled type, like say <script type="brython/javascript"/>, then brython knows to proceed without parsing / translating the code, just like if it found the code in indexedDB cache.  This gives developers the option to add better first-load performance and reduce resource duplication if needed.  While preserving the current simplicity for those who want it. 

6) Couldn't precompilation be implemented invisibly, without even needing to explicitly run a compile cycle?  The web server can compile and cache brython scripts when it detects changes to the source, then serve the compiled result.  Any significant brython app needs to be served from more than just "python -m http.server".  In which case the server can be configured to handle precompilation transparently.  Developers and end users won't see any difference at all from how things work currently.  For cases where configuring a web server to do precompiles is too much effort, brython can continue to work exactly as it does now, parsing and translating python code on the fly in each browser.

Would you be willing to support more system-level optimizations as an optional step, as long as current development simplicity is maintained? 

It would sure be interesting to develop eg a Flask app that does what you suggest, and measure the improvement on the overall speed compared to the current situation (with the indexedDB cache), but I don't have the energy to do it myself.

Just for the record, I recently switched from Firefox to Chrome for my development, and observed much slower response times. I finally discovered that it was because of the web server: those based on CPython's http.server serve JS scripts much slower on Chrome than Firefox (I couldn't understand why, but it's likely that both don't send exactly the same requests the same way); I switched to aiohttp and it solved the issue.

[Denis Migdal]

> Adding a regular file output should be trivial, and makes it easier to run separate projects.

I use something different in Brython-AOT, but else, you only need to do:

jscode = __BRYTHON__.pythonToJS(code);

const fct = new Function(jscode);

fct();

[Denis Migdal]

@Pierre: it doesn't seem that Brython is using If-Modified-Since when fetching scripts.
It might be interesting to use it in order to not download scripts if they weren't modified since their last download.

Could even be useful to be able to generate and use an import map containing the url and modification time of each Brython files.
Enabling to know with only one query which files changed, and then to only download the changes.

> It would sure be interesting to develop eg a Flask app that does what you suggest, and measure the improvement on the overall speed compared to the current situation (with the indexedDB cache), but I don't have the energy to do it myself.

Also, the generated JS is ~7x bigger than the original python source code.
So it'd depend on the network performances, on the file size, on the number of files, on the performance of the compression algorithm, etc.
And ofc it depends on indexDB speed, which depends on the speed of your hardware.
The tradeoff is hard to evaluate and might change from one user to another.

[Pierre Quentel]

Le jeu. 31 oct. 2024 à 22:00, Denis Migdal <denis....@gmail.com> a écrit :

@Pierre: it doesn't seem that Brython is using If-Modified-Since when fetching scripts.
It might be interesting to use it in order to not download scripts if they weren't modified since their last download.

This is a good illustration of why I try to avoid adding execution options : even one of the persons who know Brython best has not noticed the option "cache" which enables the use of browser cache if the server supports it. It is disabled by default but active if you set the option cache="true"

[Denis Migdal]

> This is a good illustration of why I try to avoid adding execution options : even one of the persons who know Brython best has not noticed the option "cache" which enables the use of browser cache if the server supports it. It is disabled by default but active if you set the option cache="true"

I am not familiar with the browser caching system.

It seems there are 3 interesting caching options :
- no-cache : will ask each time the server if there are changes.

- default : will not ask the server if the resource is "fresh" enough (=> this may cause update issues if one resource is updates while another is still "fresh").
- reload/no-store : do not use the cache (used in dev environment).

This being said, I have a train to catch ;)

[Ed]

This is a good illustration of why I try to avoid adding execution options : even one of the persons who know Brython best has not noticed the option "cache" which enables the use of browser cache if the server supports it.

My car has an option for emergency blinker lights.  I've never used them.  But if I crash on the side of the highway then they become priceless.

If I only added options that users need immediately then I would quickly be out of a job.

ps - I have seen the cache option before :)

though not in the category of "persons who know brython best"

--

You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/brython/CA%2BgUBLT5yc8J7WH1xWKTmHp_R1Kkiu65L99S32qQR1CZG443bQ%40mail.gmail.com.

[Ed]

I use something different in Brython-AOT, but else, you only need to do:

jscode = __BRYTHON__.pythonToJS(code);

const fct = new Function(jscode);

fct();

Thanks Denis. Say you export jscode to a file and load it in a new page. Then:

1) Is it enough to include <script type="text/javascript" src="jscode.js">? Or do you need to explicitly invoke as above, like :

jscode = doc.querySelector ('script.myjscode').innerText

new Function (jscode) ();

2) I assume brython () has to already be running on the page before starting jscode? Code is transpiled, but presumably calls brython support functions for things like importing modules.

 

[Pierre Quentel]

Le vendredi 1 novembre 2024 à 13:11:15 UTC+1, Edward Elliott a écrit :

This is a good illustration of why I try to avoid adding execution options : even one of the persons who know Brython best has not noticed the option "cache" which enables the use of browser cache if the server supports it.

My car has an option for emergency blinker lights.  I've never used them.  But if I crash on the side of the highway then they become priceless.

We must have the same model then, every time I start my car I have to decide if I want the option "emergency blinker lights" enabled or not ;-) 

[Denis Migdal]

> 1) Is it enough to include <script type="text/javascript" src="jscode.js">?

Add "defer" to the script tag and put it after the brython library so that it will run after Brython has been initialized.

> 2) I assume brython () has to already be running on the page before starting jscode?

Yes

--

You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/brython/CAF1%3DpDkbAuLkZ-AjY5y%3DvE%3DvJsqOJphFfzr5k0zr6i6p3w-cUQ%40mail.gmail.com.

[Ed]

Why does the Editor page on brython website use an old version?  It loads brython 3.8.0.

Page header includes <script src="/assets/header.brython.js"> which resolves to https://brython.info/assets/header.brython.js.

Editor page has a button to show generated js, which is very nice for testing precompiled scripts.  If it can't use latest version, then I need to use Denis's method to generate.

--

You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/brython/79be0f69-cedb-4b64-81ff-b71844d03fa6n%40googlegroups.com.

[Denis Migdal]

You can use the test page :
https://brython.info/tests/

To view this discussion visit https://groups.google.com/d/msgid/brython/CAF1%3DpDnc1-e1g6ZiGnUgJvxj3C-mLmZqo7Q7c87Nosyo0%2Br%3D3Q%40mail.gmail.com.

[Ed]

Thanks, didn't know that page.

Interesting, test page generates same code as Editor.  Now that I look again, editor also includes <script  src="/src/brython_ready.js"> which is latest version 3.13.0.  Console confirms the running version is 3.13.0.  Not sure what's going on there.  But Editor seems ok.

To view this discussion visit https://groups.google.com/d/msgid/brython/CAFr00W4ZEOx4h2ysDwzpA8MkPRyQOmbNdxYS7e49eStBUQfq-A%40mail.gmail.com.

[Denis Migdal]

A small update.

I rewrote the way I generate the JS. ast2js is now 1,4x faster.
ast2js functions are now easier to write, and more compact.

The next step would be changing the way I navigate the AST tree for my ast2conv.

This would enable me to better deduce the type of symbols declared out of order.

The .js size went from ~180kB to ~200kB.
It seems this is due to Webpack injecting lot of junk code when using imported symbols.
I could optimize it further, but this isn't something worth it now as py2ast is still, by far, the bottleneck.

Some gotcha that are quite strange:
- a += b.toString() is often faster than a += b...
- `a${b}c` is faster than ["a", b, "c"] ....

Before I used something like this (with some helper functions):
```
function toJS(this, cursor) {
      let js = "ok";
      cursor.col += 2;
      js += toJS(r`value: ${42}`, cursor);
      return js;
}
```

This caused several issues :
- I may forget to correctly update the cursor, and had to update it even when adding junk strings.
- I had difficulties to generate the correct indentation (depends on the father ASTNode that I didn't had access).
- some helpers were useful but made conditions and loops in the generation harder to handle.

I then decided to hide the cursor and the generated js code, and to manipulate with only helper functions, enabling me to facilitate things :
```
function write(this) {
     w("ok");
     wt`value: ${42}`;
}
```

I also use some special values (NL, BB, BE) to insert line breaks and to modify the indentation level.

[Ed]

I can't get precompiled brython to work.  No idea what I'm doing wrong.

Using pie chart example from brython gallery.  Copy python code to Editor (or Testing) page, use Javascript button to get compiled source.  Save as separate file pie_chart.py.js.

Page loads, brython loads, but script doesn't run.  Console gives this error:

Uncaught TypeError: Cannot set properties of undefined (setting '__name__')
    at pie_chart.py.js:7:17

Which corresponds to this compiled code, that doesn't even look like valid js to me:

var $B = __BRYTHON__,

    _b_ = $B.builtins,

    locals_exec = $B.imported["exec"],

    locals = locals_exec,

    frame = ["exec", locals, "exec", locals]

var __file__ = '<string>'

locals.__name__ = 'exec'   // <--- LINE 7

locals.__doc__ = _b_.None

Doesn't matter whether I include the compiled js directly with <script src="pie_chart.py.js">, or if I save the text to a var, make new Function (), and call that.  Same error either way.

Attached are the html page, python source, and compiled python-to-js (named pie_chart.py.txt because even in 2024 email clients are so braindead stupid that gmail refuses to attach a file named *.js for security reasons).

Steps to reproduce

1. Rename pie_chart.py.txt to pie_chart.py.js
2. Download brython.js and brython.stdlib.js (or point to CDN versions)
3. Open the page with python -m http.server

To view this discussion visit https://groups.google.com/d/msgid/brython/CAFr00W4ZEOx4h2ysDwzpA8MkPRyQOmbNdxYS7e49eStBUQfq-A%40mail.gmail.com.

[Denis Migdal]

You need to prefix your code with :

__BRYTHON__.imported["exec"] = {};
__BRYTHON__.frames_stack = [];

Cf Brython-AOT code.

[Ed]

Thanks.  With your prefix, now I get a different error:

Uncaught Error
    at Object.$factory (brython.js:1:230223)
    at brython.js:1:558486
    at Object.$getitem (brython.js:1:86207)
    at pie_chart.py.js:30:26

where line 30 is :

locals_exec.panel = $B.$getitem(locals_exec.document, 'panel',[8,17,24])

Page has a tag <g id="panel"> inside <svg>.   No idea what's wrong.  It shouldn't be this hard.

To view this discussion visit https://groups.google.com/d/msgid/brython/820bc463-456e-4417-8cf6-aade3a170d51n%40googlegroups.com.

[Denis Migdal]

You forget to add the "defer" attribute to your script tags.

Here the JS is executed synchronously, so the DOM hasn't finished loading, i.e. #panel doesn't exist yet.

To view this discussion visit https://groups.google.com/d/msgid/brython/CAF1%3DpDk-siH_DOkeT%3DPgcJXGCQJWmCOXOrLEEUXWu8P9fc7SjQ%40mail.gmail.com.

[Ed]

You're right, I did.  You mentioned that before.  I'm a dope.

Now it gives a new error:

Uncaught ReferenceError: $varB is not defined

    at pie_chart.py.js:41:3

which is:

$varB.set_lineno(frame, 21)

Remote error debugging is fun!  It's like putting my punchcards in a batch job and waiting overnight for a printout that says I forgot a comma. :)

To view this discussion visit https://groups.google.com/d/msgid/brython/CAFr00W4CRpDj54uo9bueh10QJxutiuWBX8i%3DgRVZG9xPpUvZ1Q%40mail.gmail.com.

[Denis Migdal]

Did you edit the generated JS file ?

It should be $B, not $varB

To view this discussion visit https://groups.google.com/d/msgid/brython/CAF1%3DpDmpOKDZDALBVTGdAXRGypHvTO-qpX8Wen6KwqDufpJGqA%40mail.gmail.com.

[Ed]

Good catch.  Not intentionally.  I must've accidentally added it when trying the new Function approach and wrapping the code with var jscode = `...`.  Editing slip.

It runs now.  Thanks for the help.  Normally I can debug my own python and js, but a bit lost with all this generated code I don't understand.

To view this discussion visit https://groups.google.com/d/msgid/brython/CAFr00W4c%3Dj5Fqd0c20jg3XKqLwbDgbFPUFyYzxv%2B2yXCHfsu-g%40mail.gmail.com.

[Ed]

It would sure be interesting to develop eg a Flask app that does what you suggest, and measure the improvement on the overall speed compared to the current situation (with the indexedDB cache)

I posted some initial measurements on github if anyone is interested:

https://github.com/brython-dev/brython/discussions/2509

Probably better to continue this topic there.

[Denis Migdal]

Hi,

Long time no see ;)


A few updates. I tested DOP principles using SBrython and got an overall -20% execution speed.

By merging the tests (200+ lines in total) into a single file, `genFct` costs almost nothing (0.3ms), while `ASTConv` and `ast2js` cost ~6.5ms.
I'd like it to be anecdotal compared to `genFct`, but this is 20x slower. Well, I guess I can't beat the native browser parser xD.

Brython `py2ast` is ~5x time slower, but I think I'll be able to optimise it once I'll rewrite it.

Though I think I will be able to increase the performances by:
- building SBrython in production mode (minified).
- removing some checks and code positions tracking in production mode.
- a few miscellaneous optimisations.

I'll have also to rewrite my editor to print better stats.

I do not know the standard file length, but "tifffile.py" has ~24,000 lines, and is loading in the browser in ~56ms.
Could be great if py2js was ~5ms...

Currently, it is ~1,500ms for Brython. I can't compare with SBrython (lot of features not yet implemented).
For 24,000 lines, SBrython should be 780ms max for `ASTConv` + `ast2js` and 6,500ms in total, while being nearly 2x faster than Brython...
"tifffile.py" may be doing a different kind of operation than what I have currently in my test suite...

[Denis Migdal]

Hi,

Discovered nice things theses last days, some that I will test this WE (some expect some news), and other things...

I discovered "AssembyScript" : https://www.assemblyscript.org/ , in short JS with type hints (like Python have type hints), that can be converted to WASM.

The advantages are multiples :
- at least x2 execution speed ;

- code is optimized immediately, not after a few run of the functions (and optimizations are more predictable) ;

- we can manage allocations ourselves, use real arrays, use pointers, etc. ;

Currently, py2ast seems to be Brython's bottleneck.
Therefore, I wonder if someone would be interested to try to annotate the current brython JS code with type hints, and to change the generated AST output format to mine (easier to share through SharedArrayBuffer). This in order to check if we have indeed a significant speed increase.
Note: we may have issues with Regexes.

ast2js is harder to write in WASM as the JS code string generation would need special structures that may be non-trivial.

[Denis Migdal]

Pushed the new version of my editor (JS code print is currently bugged, AST print is not quite readable).

I'm now printing the time it would take for 100k tokens (tifffile is ~150k tokens).

I merged all the tests, putting then each in a function (so lots of function creations and calls).

In SBrython, astProc is my old ASTConv with type deduction, checking, etc. In Brython, this is the symtable construction, future, etc (it was previously integrated into ast2js).

As always Brython's py2ast is the bottleneck by far. I'm pretty confident we can greatly speed it up.

Next week-end, I'll work on a production version :
- without indentation ;
- without some optional checks ;
- without JS/Py code position tracking ;

I'm curious about how much it could increase performances.

I'll have also some parts to optimise :
- my module system (I want to de-correlate ASTConv and ast2js parts) ;
- my type representation in my type system ;
- optimise values storage (I can store some values into my Float64Array) ;
- review my JS code generation ;
- the way I handle operators.

I think I will have quite a few week-ends of work ^^.

Next steps, I think it would either be adding new features to SBrython, or to implement my own parser (now that I can access tokenize).

I could implement the parser in WASM, but I think this would be too much work for now.

[Denis Migdal]

Some quick updates:

- astProc 3.180ms -> 2.060ms (x1.54 faster than previously - x1.22 faster than Brython)

- ast2js 2.300ms -> 1.860ms (x1.24 faster than previously - x3.43 faster than Brython)

I introduced a "production" mode that removes some checks, the code position tracking, etc.
Production version weights 40kB (12kB once compressed).

I fixed some logical errors (conditions that were always true/false).
I rewrote the way I write the JS code. I modified how I define my types to be closer to Python.

I still have room for performances improvement, mainly on operators, and functions argument parsing.
However, I think it would be better to wait until I rewrite py2ast as it is now responsible of 97.7% of the total execution time...

After cleaning some parts of the code, I think the next step would be to start adding new features again to pass more of Brython's unit tests.
As I have now a "production" mode I can also add some standard Python error message... maybe.

[Denis Migdal]

Great news, I started to implement my own (naïve) parser.
I believe I could aim for a 30x speed increase (ofc while taking some shortcuts). We'll see what we'll have at the end ;)
(I also started to support some classes features, but this isn't finished).

I had to slightly transform my AST, to be able to more easily add children and swap nodes.
Indeed, when parsing we may not initially know how many child a node has (e.g. how many lines a Body has), or we may have to swap/reorder children (e.g. for operator priorities).

The code feels slightly slower for astProc and ast2js.
Well, with my own parser, the brython to sbrython AST conversion would be useless, and I'll be able to extract the exact informations I need for my generation.
Moreover, I did some quick non-optimized adaptions, and still have some possible optimizations (fonctions definition/call, operators, etc).

From my work, I strongly believe that Brython:
- needs a development and a production mode: in production, we need performance and smaller library size. We also know that some syntax errors will not occurs. (once SBrython is finished, it may assume the production mode role -?-).
- should represent its data in a JS-friendly way, and only do the Python specific-things inside its manipulations. This would prevent pyobj <-> jsobj conversions issues, while facilitating JS <-> Python interactions (e.g. developing a JS library). SBrython seems to demonstrate that this is possible, but I still have a lot of unit test to pass.
- should be able to produce ES6 modules, that could then be integrated into standard Web development pipelines (minification, uglyfication, module concatenation, tree shaking, etc.).

[Denis Migdal]

9/36 type of AST node parsing implemented for the naive parser.
Honestly, it is currently easier than I though.

On very small code, I'm ~15x faster for py2js, ~20x faster for py2ast.
On bigger code I should be even faster. Though, currently I can't test on the Brython unit test suite, not enough AST node parsing implemented.
When implementing more, it should decrease a little the performances.

Currently, I think this is fast enough to not need the cache.

I'm also quite curious about the performances we could achieve in WASM.

See you next week for more news ;)

[Denis Migdal]

Great news everyone, I just passed 132 unit tests with the new parser (had to disable some of them).


Currently, the script is 36kB (11kB once compressed), which is quite small. It will ofc increase once I'll add standard Python classes/methods.

Execution is still 5.5x faster.
Py2JS is 13x faster, and py2ast 16x faster.

I still have some ideas to improve performances, but I also need to pass more unit tests to have more stable performances results.


Be reminded that SBrython is for production code, hence:
- you might have infinite loop for invalid code.
- you don't have all the Python/Brython error message.
- some edge cases aren't implemented.
- you may need to specify variable types.

Import implementation will be quite a challenge as I'll need to store/register type informations (which is a circular structure).

Classes are still missing, but once done I think I'd have most of the core work, only needing to implement standard structures/methods/functions.

[Denis Migdal]

Hi,


I will soon create flags to manipulate SBrython output.

- compatibilities levels : to choose the level of compliance to the Python standard (at the cost of code readability, speed, size).

- output formats : to choose between ES6 module, runtime module, TypeScript ES6 module, etc.
- of course __DEBUG__ flag.

For compatibilities, I aim at having ~3 main compatibilities level :
- JS compliant : for speed & educational purpose : produce clean, pure JS code, without runtime. i.e. with LOT of differences with the python norm (though most of student code doesn't relies on the Python nuances).

- static : some runtime code is added to be a little more compliant with Python. JS code is a little less clean. But should still be quite quick.

- Python compliant : produces lot of runtime code, not performant in speed or size.

Each of this version will enable/disable a set of minor flags (e.g. "keep operations order", "enable kw arguments", etc). The minor flags will be used for debug/dev purposes, and will not be directly tested/officially supported by SBrython. Their goal is to allow to easily add/modify the main compatibilities levels.

Having lot of flags isn't an issue as I just need to test all combinations on the unit tests (enabling some unit test for some flags).
I estimate that I could test ~3600 possibilities in 1 hour, so ~12 binary flags.

With what plan actually, I should have ~3x3x2 = 18 possibilities, so tested in ~18 seconds (ofc can be faster if tested in several threads).

Concretely, I should have very soon a first AoT version with export support.

Cordially,