Understanding the core of Brython
158 views
Skip to first unread message
pi
Sep 5, 2015, 8:19:15 AM9/5/15
to brython
Olemis hinted at this the other day I believe.
It would be very nice for someone that understands Brython inside out to give a brief walk-through that digs into the internals and helps us understand the basic structure --how it does its magic. i.e. targeted at someone that understands JavaScript and Python.
Maybe start with a minimal "hello world" style program, and then go into exactly how Brython operates on this program, maybe linking to appropriate line numbers on GitHub (although that's tricky because future edits invalidate them).
I think this would encourage people to contribute development.
As it is, it's rather daunting.
Maybe Pierre would care to say something?
I would also love to read about how the project originally took form, and how it has evolved.
π
Pierre Quentel
Sep 9, 2015, 6:17:49 AM9/9/15
to brython
That's a very good idea. I used to maintain implementation notes in the documentation in earlier versions, but they were difficult to maintain (things change at each new version).
I have started writing a general overview (see below, in raw markdown) : is this helpful ? If so I will develop it and add examples of translation. I think Brythonista (my blog) would be a good place for that.
- Pierre
How Brython works
=================
Translation from Python to Javascript
-------------------------------------
A typical Brython-powered HTML page looks like this :
```
<html>
<head>
<script src="/path/to/brython.js"></script>
</head>
<body onload="brython()">
<script type="text/python">
...
</script>
</body>
</html>
```
brython.js exposes 2 names in the global Javascript namespace : `brython`
(the function called on page load) and `__BRYTHON__`, an object that holds all
internal objects needed to run Python scripts.
The function `brython()` inspects all the scripts in the page ; for those that
have the type `text/python`, it reads the Python source code, translates it to
Javascript, and runs this script by a Javascript `eval()`.
If the `<script>` tag has an attribute `src`, an Ajax call is performed to
get the content of the file at the specified url, and its source is converted
and executed as above.
The translation to Javascript takes the following steps :
- a tokenizer reads the tokens in the source code and pass them to an automat
that builds an astract tree for the code, or raises `SyntaxError` or
`IndentationError`.
- this tree is transformed (nodes are added or modified) to translate some
single Python statements in a number of Javascript statements.
- if the debug level is set, additional nodes are added to update an internal
object that is set to the current script name and line number.
- the transformed tree supports a method `to_js()` that returns the Javascript
code.
All this is done in the script __py2js.js__ :
- function `brython()` is the last one in the script
- translation is done by function `py2js()`
- this function calls the tokeniser : function `tokenize()`
- the tokenizer builds a tree made of instances of the class `$Node`
- an instance of `$Node` is created for each new statement, and a context is
created for the node
- new tokens generally change the state of the context by a call such as
`context = transition(context, token_type, token_value)`
- context is an instance of one of the classes defined in the script, whose
name starts with `$` and ends with `Ctx` : for instance, when the tokenizer
encounters the keyword `try`, the function `transition()` retuns an instance
of `$TryCtx`
Builtin objects
---------------
Among the attributes of `__BRYTHON__`, all the built-in Python names (classes,
functions, exceptions, objects) are stored, usually with the same name : for
instance, the built-in class `int` is stored as `__BRYTHON__.int`. Only names
that conflict with Javascript naming rules must be changed, eg `super()` is
implemented as `__BRYTHON__.$$super`.
Implementation of Python objects
--------------------------------
Python strings are implemented as Javascript strings.
Python lists are implemented as Javascript arrays.
Python integers are implemented as Javascript numbers if they are in the range
of Javascript "safe integers", ie in the range [-(2**53-1), 2**53-1] ; outside
of this range they are implemented with an internal class.
Python floats are implemented as instances of the Javascript `Number` class.
All other Python classes (builtin or user-defined) are implemented with 2
Javascript objects :
- a function to create instances
- an object that holds the class attibutes and methods
Name resolution
---------------
A Python program is divided in blocks : modules, functions, classes. For each
block, Brython defines a Javascript variable that will hold all the names
bound in the block (we call it the "block names object").
Based on lexical analysis, including the `global` and `nonlocal` keywords, it
is generally possible to know in which block a name is bound. It is translated
as the attribute of the same name of the block names object.
The only case when the block can't be determined is when the program imports
names by `from some_module import *`. In this case :
- it is impossible to know if a name like `range` referenced in the script is
the built-in class `range` or if it was among the names imported from
`some_module`
- if a name which is not explicitely bound in the script is referenced,
lexical analysis can't determine if it should raise a `NameError`
In this case, the name is translated to a call to a function that will select
at run time the value based on the names actually imported by the module, or
raise a `NameError`.
Execution frames
----------------
Brython handles the execution frames in a stack. Each time the program enters
a new module or a new function (including lambdas and comprehensions),
information about the global and local environment is placed on top of the
stack ; when the function or module exits, the element on top of the stack
is removed.
This is done by inserting calls to the internal functions `enter_frame()` and
`leave_frame()` in the generated Javascript code.
The stack is used for instance by built-in functions `globals()` and
`locals()`, and to build the traceback information in case an exception is
raised.
I have started writing a general overview (see below, in raw markdown) : is this helpful ? If so I will develop it and add examples of translation. I think Brythonista (my blog) would be a good place for that.
- Pierre
How Brython works
=================
Translation from Python to Javascript
-------------------------------------
A typical Brython-powered HTML page looks like this :
```
<html>
<head>
<script src="/path/to/brython.js"></script>
</head>
<body onload="brython()">
<script type="text/python">
...
</script>
</body>
</html>
```
brython.js exposes 2 names in the global Javascript namespace : `brython`
(the function called on page load) and `__BRYTHON__`, an object that holds all
internal objects needed to run Python scripts.
The function `brython()` inspects all the scripts in the page ; for those that
have the type `text/python`, it reads the Python source code, translates it to
Javascript, and runs this script by a Javascript `eval()`.
If the `<script>` tag has an attribute `src`, an Ajax call is performed to
get the content of the file at the specified url, and its source is converted
and executed as above.
The translation to Javascript takes the following steps :
- a tokenizer reads the tokens in the source code and pass them to an automat
that builds an astract tree for the code, or raises `SyntaxError` or
`IndentationError`.
- this tree is transformed (nodes are added or modified) to translate some
single Python statements in a number of Javascript statements.
- if the debug level is set, additional nodes are added to update an internal
object that is set to the current script name and line number.
- the transformed tree supports a method `to_js()` that returns the Javascript
code.
All this is done in the script __py2js.js__ :
- function `brython()` is the last one in the script
- translation is done by function `py2js()`
- this function calls the tokeniser : function `tokenize()`
- the tokenizer builds a tree made of instances of the class `$Node`
- an instance of `$Node` is created for each new statement, and a context is
created for the node
- new tokens generally change the state of the context by a call such as
`context = transition(context, token_type, token_value)`
- context is an instance of one of the classes defined in the script, whose
name starts with `$` and ends with `Ctx` : for instance, when the tokenizer
encounters the keyword `try`, the function `transition()` retuns an instance
of `$TryCtx`
Builtin objects
---------------
Among the attributes of `__BRYTHON__`, all the built-in Python names (classes,
functions, exceptions, objects) are stored, usually with the same name : for
instance, the built-in class `int` is stored as `__BRYTHON__.int`. Only names
that conflict with Javascript naming rules must be changed, eg `super()` is
implemented as `__BRYTHON__.$$super`.
Implementation of Python objects
--------------------------------
Python strings are implemented as Javascript strings.
Python lists are implemented as Javascript arrays.
Python integers are implemented as Javascript numbers if they are in the range
of Javascript "safe integers", ie in the range [-(2**53-1), 2**53-1] ; outside
of this range they are implemented with an internal class.
Python floats are implemented as instances of the Javascript `Number` class.
All other Python classes (builtin or user-defined) are implemented with 2
Javascript objects :
- a function to create instances
- an object that holds the class attibutes and methods
Name resolution
---------------
A Python program is divided in blocks : modules, functions, classes. For each
block, Brython defines a Javascript variable that will hold all the names
bound in the block (we call it the "block names object").
Based on lexical analysis, including the `global` and `nonlocal` keywords, it
is generally possible to know in which block a name is bound. It is translated
as the attribute of the same name of the block names object.
The only case when the block can't be determined is when the program imports
names by `from some_module import *`. In this case :
- it is impossible to know if a name like `range` referenced in the script is
the built-in class `range` or if it was among the names imported from
`some_module`
- if a name which is not explicitely bound in the script is referenced,
lexical analysis can't determine if it should raise a `NameError`
In this case, the name is translated to a call to a function that will select
at run time the value based on the names actually imported by the module, or
raise a `NameError`.
Execution frames
----------------
Brython handles the execution frames in a stack. Each time the program enters
a new module or a new function (including lambdas and comprehensions),
information about the global and local environment is placed on top of the
stack ; when the function or module exits, the element on top of the stack
is removed.
This is done by inserting calls to the internal functions `enter_frame()` and
`leave_frame()` in the generated Javascript code.
The stack is used for instance by built-in functions `globals()` and
`locals()`, and to build the traceback information in case an exception is
raised.
Vinicius Assef
Sep 9, 2015, 9:15:58 AM9/9/15
to bry...@googlegroups.com
Too good, Pierre! :-)
—
Vinicius.
—
Vinicius.
--
You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.
To post to this group, send email to bry...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/brython/f2358bb8-1140-4db4-8e5a-a3a29ca4f593%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
pi
Sep 10, 2015, 5:18:59 PM9/10/15
to brython
This is great: exactly what I was imagining!
Surely the best place for this would be in the GitHub repo, in the root folder... that way it would be easy to accrete external contributions. Also, that is the first place people would look. Also, GitHub renders markdown.
π
Pierre Quentel
Oct 28, 2015, 1:17:49 PM10/28/15
to brython
I have added a page in the GitHub wiki : https://github.com/brython-dev/brython/wiki/How-Brython-works. It is referenced in the section "Documentation" of the GitHub home page.
Kiko
Oct 28, 2015, 1:31:32 PM10/28/15
to bry...@googlegroups.com
--
This is very complete.
Thanks, Pierre.
You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.
To post to this group, send email to bry...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/brython/9939951f-447b-4610-adea-5f9f6ebe1bf2%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages