RBrython

[Denis Migdal]

Hi,

As you can see, I'm back ;)

I'll talk about several things, so don't hesitate to pick what interest you ;).

Previously, I demonstrated (among other things) that we can produce small, clean, and fast JS by using types known at compile time. However as all types need to be known at compile time, we need a strong deduction type system.

I also used DOP principles to achieve good performances during parsing and emission. However this quickly become a pain when you have complex types (unions, type guards, etc). A pain to achieve performances in a part I'll argue isn't the priority.

Now I want to test other ideas with RBrython, this time with a dynamic type system (i.e. not known at compile time). I'll work time to time on it as a pet project, when I want to do something for a change.

I'll try to document as much as possible, as well as documenting similar Brython features. Offering some suggestions and opinions. It'll prevent me from flooding github issue.
E.g. : https://github.com/denis-migdal/RBrython/blob/master/docs/runlib/index.md

In this new project, I want to clearly segregate different parts:
- parser (py2ast using brython)
- emitter (ast2py)
- runlib : helpers required to run the JS code.
- corelib : implementation of global python symbols (e.g. int, range, etc).
- stdlib : standard python library.

Indeed, I think that parser, emitter and runlib aren't that big, and can be implemented quite easily (maybe I'll soon change my mind about it xD). However corelib and stdlib are an huge code base. This means that if they depends on the emitter or on the runlib, each changes e.g. on functions calls, impact all of this huge code base.

I recently read "clean code", "clean architecture", and "refactoring" books, it was quite interesting. And indeed, keeping things segregated is very important.

I want to write corelib in Python, using special emitter rules to generate the JS. Then, if something changes, we just have to modify the emitter rules. For example:

class int:

def __new__(cls) :

return 2

def __add__(self, b: int) -> int:

return __JS_ADD__(self, b) # type: ignore

I think this will make contributions easier on corelib and stdlib, without having to know too much on Brython specifics. Also, if something changes in emitter/runlib, this shouldn't impact too much of the code base.

For performances, I can then have 2 more processes :
- checker : assert types at compile time.
- optimizer : modify the AST.

When optimizing, instead of trying to modify the emitter rules, I'll change the ASTNode type and add new emitter rules. I can then, e.g. inline some function calls.

Ofc, I still want no JS-Python conversions. I'll argue that we should manipulate vanilla JS data, and that Python specific behavior should be implemented in the Python operations.
Making it easier to write JS libs for Brython to achieve native performances.

I'll keep you updated on my progress ;)

You can read more on:
https://github.com/denis-migdal/RBrython/

[Denis Migdal]

I can now implement types in Python:

from functools import singledispatchmethod

from types import NotImplementedType

class int:

def __new__(cls, o: object, /) -> int:

return type(o).__int__(o) # type: ignore

@singledispatchmethod

def __add__(self, _: object, /) -> int|float|NotImplementedType:

return NotImplemented

@__add__.register

def _(self, b: int, /) -> int:

return __JS_OPI__(__JS_ADD__, self, b) # type: ignore

The big advantage is that this is totally independent on how you convert python code into JS. Meaning that you can change how you implement Python classes, how you implement function calls, etc. this code doesn't need to change.

When creating a literal, an internal value is stored in the instance (using a symbol). You can then manipulate internal values using special methods:
- __JS_GET_IVALUE(self)        # get the internal value
- __JS_SET_IVALUE(self, val) # set the internal value

You can also perform JS operations on theses internal values, e.g. __JS_ADD__(a, b) to perform the JS addition.
__JS_OPI__(op, a, b) is equivalent to op( __JS_GET_IVALUE(a), __JS_GET_IVALUE(b) ).

I didn't optimized it yet, theses special functions are currently called like normal JS functions.
However, we can very easily inline it during code emission (ast2js). Replacing the "__JS_OPI__" call directly by "self[IVALUE] + b[IVALUE]".
 
Other things could be optimized, but it will only require to change emission rules.
@Pierre: do you think this could be usable in Brython ?

note: @singledispatchmethod is used to facilitate future optimizations (easier to inline and assert types).

[Denis Migdal]

I really have difficulties stopping myself when I'm on something ^^.

A possible optimization that could also help Brython.
The line "if 1 == 1:" is written as:

if($B.set_lineno(frame, 1) && $B.$bool($B.rich_comp('__eq__', 1, 1)))

@pierre is there a reason to use "$B.set_lineno() && $B.bool() " instead of "$B.set_lineno() , $B.bool() " ?

Before emitting the JS, you can optimize it by splitting the "if" and "==" ASTNodes:
- "if" becomes "if_raw" + "to_bool".
- "cmp" becomes "from_bool" + "cmp_raw" (depends on how exactly this is implemented).

Then the operations are now : if_raw + (to_bool + from_bool) + cmp_raw.
to_bool and from_bool cancels each other.

By splitting some AST node before emitting the JS, you can remove some useless operations. I think this likely occurs when the node convert its input or output data.

Instead of creating new AST node, maybe some properties like node.input_conversion / node.output_conversion could be added to the node (?).


I started to list some possible optimization in (some of which Brython is already doing) : https://github.com/denis-migdal/RBrython/blob/master/docs/optimizer/index.md

Won't be able to test them yet, other things to test before that ^^.

[Pierre Quentel]

@pierre is there a reason to use "$B.set_lineno() && $B.bool() " instead of "$B.set_lineno() , $B.bool() " ?

The only reason it that at the time I wrote it I didn't know about the ',' operator in Javascript ;-)

Does it make a difference on performance ?

 

Before emitting the JS, you can optimize it by splitting the "if" and "==" ASTNodes:
- "if" becomes "if_raw" + "to_bool".
- "cmp" becomes "from_bool" + "cmp_raw" (depends on how exactly this is implemented).

Then the operations are now : if_raw + (to_bool + from_bool) + cmp_raw.
to_bool and from_bool cancels each other.

I don't understand what you mean here, can you develop a little more ? 

[Denis Migdal]

> Does it make a difference on performance ?

According to deno bench, the condition is ~10% to ~35% faster with ",".
However, I do not know if it will be really significative on a whole Brython code (as you would have other expensive operations).

One good thing with "," is that you clearly show your intent, by explicitly stating that set_lineno isn't relevant for the condition.
It'll help third parties to understand the generated code.

> I don't understand what you mean here, can you develop a little more ? 

Maybe I'll come back once I'd implement it (I have a few things I want to implement first in my todolist). 

The general principle is that sometime you may perform a conversion, then perform the opposite conversion, e.g. f⁻¹(f(x)). In such case you can drop the conversion and directly use "x".

My initial though was to implement such optimization by manipulating the AST.
But maybe we could have a system where each node explicitly states what it expects and what it returns, and write the required conversions when needed. I think I'll need to think more about it, and to implement it to see if it is easy to do.

[Denis Migdal]

@Pierre: I took a look at pyobj2jsobj and jsobj2pyobj.

It seems that you could remove all conversions (which could remove quite a thorn), if it wasn't for float/int.

One of the two need to be implemented as JS number and the other as something else.
For this something else, by defining `valueOf()`, you should remove the need for pyobj2jsobj conversion.

I'd argue that float should be implemented as JS number to avoid the hacking JS number conversion into float or int depending on whether it has decimal or not. Which could lead to unexpected behaviors. With that you should remove the need for jsobj2pyobj conversion.

Now how to implement int ?

bigint is painfully slow. JS engines still hasn't optimized it and it is still 4x to 40x slower on small int.
But it is the easiest way to proceed (and maybe is faster on bigger int). In SBrython I used bigint and it was still quite fast overall (though I cheated a little).

If it really is too slow, you could implement small int it as {value: X, valueOf(){ return this.value } } (as you are currently doing for float).
- Accessing value surprisingly has near native performances.
- it seems adding valueOf doesn't increase the creation cost if you use a class:
```
class int {
    constructor(i) { this.value = i }
    valueOf() { return this.value }
}
new int(2);
```

If you make the class inherit from `Number`, the creation is 2x slower.

[Denis Migdal]

Some updates:

- documentation rewritten ( https://github.com/denis-migdal/RBrython/tree/master?tab=readme-ov-file#rbrython-poc ).

- new production mode (remove asserts, __debug__ set to false ~= -OO Python option).

- Engine and Runner classes, with a Brython version ( https://github.com/denis-migdal/RBrython/blob/master/docs/engine/index.md )

- AoT compiler, can also be used with Brython.

- for performances ~2x faster than Brython so it seems the Python implementation of builtins doesn't cost too much.

Honestly I kind of like this architecture. This makes customization quite easy.
We can easily change the builtins and macro implementations, use another parser or emitter, etc.

In the corelib, I replaced the @dispatch, by a class pattern matching which is a little nicer to use:

from types import NotImplementedType

from RBM import __JS_AS_NUMBER__, __JS_OP__, bigint

class int(bigint): # h4ck to remove PyLance errors

def __add__(self, o: object, /) -> NotImplementedType|int:

match o:

case int (): return __JS_OP__(self, "+", o)

case _ : return NotImplemented

Next work, the sourcemap, enabling to use the browser debug tools with RBrython.
Unfortunately, I can't do it with Brython as Brython doesn't save the generated JavaScript code position when emitting the JavaScript.

Then, the only thing remaining will be unitest, unitest, unitest xD.

I also plan to add some export mode (e.g. to produce ES6 modules), and some compatibility/optimisations modes at a later date.

[Denis Migdal]

Some quick updates:
- a very basic type checker system.
- new options to state how you want your Python module to be exported (as a Brython module ? as an ES6 module ? as a function ?).
Not properly implemented for Brython, but I can do it in few minutes if somebody needs it.
- new options to handle the performances/compliance tradeoff you wish to have (disabled -no opti-, safe -safe opti-, unsafe).
- use of functions inside emission handlers that I can substitute. e.g. boolean coercion is used in several kind of AST node. I can then provide several versions on how to write this boolean coercion in the emitted JS.
- a page for the kernel implementation status: https://github.com/denis-migdal/RBrython/blob/master/docs/status/index.md
- a page to gather possible optimizations: https://github.com/denis-migdal/RBrython/blob/master/docs/optimizers/index.md
- editor improved (JS code syntax highlight + indentation).
- finally, I think we may be able to use ES6 (with some tweaks), as long as we do not use the JS constructor.

One interesting thing is the optimization page where I explain when we can't write "normal JS", why, and the conditions required to write "normal JS". Most of it requires a form of type checking. One of them doesn't : you don't need boolean coercion after a comparison (will always produce a boolean).

The remaining thing is to:
- implement (or test) missing feature
- pass Brython unit test
- improve the type checker and optimisations.