Progress being made on pyjaco

[Dexter]

Hi everyone,

It's been quiet on the mailinglist, but not as quiet on the project.

A long while ago some plans were laid down in order to vastly improve the compiler and project.

Samuel, Christian and I have defined some basic architecture details (which can be found here: https://docs.google.com/folder/d/0BxAPrjqUjBdZaEFUYVpXYkVqYkE/edit)

With that design I started implementing (from the ground up) and started publishing that code to: (https://github.com/neoel/pyjaco) 

The new code base is in the sub directory pyjaco.new

I have implemented an AST reader, converting AST to IST.

Started the basis of the transformers. 

Implemented a pretty (not entirely) complete python writer.

So I have achieved a python to python translator. Still working on the javascript part (which requires some transformation steps.).

I post this as Christian requested. We've been talking about it quite privately without any possible feedback from you guys. That why I want this discussion to continue on the mailing list.. 

Here will be a long line of mails. If you have any question regarding implementation or anything, don't hesitate to ask.

Greetings, Dexter Esselink.

__________________________________________________________________________________________

Dexter	Nov 29 (4 days ago)
to Christian, Samuel

Hi Guys,

To put some things on paper, and discuss this with you. Ill send some thoughts.

Yesterday we discussed the IST, and a bit on what steps to take. 

We concluded that the IST should be able to represent either python and javascript. 

To make the ist more consise/small, we thought that operators could be bundled in one IST node; called `operator` with one property, a string determining the operator in the code. 

This is still a bit troublesome imo, since we would have to do string comparisons/mappings to how this should be represented in python or javascript: (&& is not and, enz. how to deal with is/is not).

And it means we would have to map it from ast node to string in the reader, from string to behaviour in the transformers and from string to string in the writers. 

So I think we are best off keeping the complete set of AST nodes as IST. And making the functions deal with those issues. 

In that way the write_And function in the writers can do the thing they need to do (just return a simple string.

I got a question for you. In the design we have 3 steps drawn out for the transformers. 

__fication, typeification and sematification. 

the first one I understand. But I forgot what the others are supposed to do.

Can you help me out there? 

Thanks, Dexter

__________________________________________________________________________________________

Samuel Ytterbrink	2:03 PM (1 hour ago)
to Christian, me

Hi!

Let the operators be a object which have both the python string repr and the JavaScript repr of the operator, make them static so it's easy to change.

The typification i think was the JavaScript/python type annotation (which actually change the whole node) so that the next steps know if it should do any pythonic transformations to the value or the other way around.

I hope I'm right with this.

__________________________________________________________________________________________

Christian Iversen	2:06 PM (1 hour ago)
to me, Samuel

On 2012-12-03 13:58, Dexter wrote:

I know you are busy. But I have the feeling you didn't see this email.
So i'll send it again.

I found it! Accidentally overlooked it. My bad :)

By the way, we might as well have this kind of discussion on the mailing list. It's not really a secret, and it shows everybody that something is happening. If you agree, I will forward your notes to the list (or you can do it, so the sender is you)

    To put some things on paper, and discuss this with you. Ill send
    some thoughts.
    Yesterday we discussed the IST, and a bit on what steps to take.

Good with notes :)

    We concluded that the IST should be able to represent either python
    and javascript.
    To make the ist more consise/small, we thought that operators could
    be bundled in one IST node; called `operator` with one property, a
    string determining the operator in the code.
    This is still a bit troublesome imo, since we would have to do
    string comparisons/mappings to how this should be represented in
    python or javascript: (&& is not and, enz. how to deal with is/is not).
    And it means we would have to map it from ast node to string in the
    reader, from string to behaviour in the transformers and from string
    to string in the writers.

Well, mapping string-to-string is trivial with a dictionary. Mapping class-to-class is a bit more tricky, in my opinion. Yes, you can do it with a dictionary, but it seems somehow less clean to do it polymorphically. Remember, that "and" and "&&" are indeed not the same, but they are both represented by the OP_AND ("&&") operator type. The specialization that turns python into javascript changes OP_AND into GETATTR("__and__") anyway, so this will not be seen by the javascript writer.

    So I think we are best off keeping the complete set of AST nodes as
    IST. And making the functions deal with those issues.

I still think that a unified op type would be better, but either way is fine. I can see arguments both ways.

    In that way the write_And function in the writers can do the thing
    they need to do (just return a simple string.

    I got a question for you. In the design we have 3 steps drawn out
    for the transformers.
    __fication, typeification and sematification.
    the first one I understand. But I forgot what the others are
    supposed to do.

    Can you help me out there?

Well, yes, the __fication is obvious. Typeification is another pretty simple step, where we need to turn all literals into python literals. For example, 5 becomes $c5, 42 becomes $PY.int(42) (since it does not have a pre-defined constant), "foo" becomes $PY.str("foo"), and so on.

Finally, semantification is where we change classes, methods and control structures to conform to python semantics. For example, the exception handling is vastly different, and if (foo) needs to be if (js(foo)) and so on. This is what currently represents the bulk of the compiler code.

__________________________________________________________________________________________

Christian Iversen	2:09 PM (1 hour ago)
to Samuel, me, Christian

On 2012-12-03 14:03, Samuel Ytterbrink wrote:

Hi!

Let the operators be a object which have both the python string repr and
the JavaScript repr of the operator, make them static so it's easy to
change.

No, please don't :-)

Remember that the IST does NOT represent any particular syntax. As such, there is no "python repr" or "javascript repr" of it. The "and" node simply represents the "idea of and", not a particular operator. The transformation strages makes python "and" turn into javascript code, by means of simulating the semantics.

So it should simply have an .op field with the op type, and that's it.

Unless I misunderstood you?

The typification i think was the JavaScript/python type annotation
(which actually change the whole node) so that the next steps know if it
should do any pythonic transformations to the value or the other way around.

I hope I'm right with this.

Almost, I explained it in the other mail :)

__________________________________________________________________________________________

Dexter	2:35 PM (1 hour ago)
to Christian

On 2012-12-03 14:03, Samuel Ytterbrink wrote:

Hi!

Let the operators be a object which have both the python string repr and

the JavaScript repr of the operator, make them static so it's easy to

change.

No, please don't :-)

Remember that the IST does NOT represent any particular syntax. As such, there is no "python repr" or "javascript repr" of it. The "and" node simply represents the "idea of and", not a particular operator. The transformation strages makes python "and" turn into javascript code, by means of simulating the semantics.

So it should simply have an .op field with the op type, and that's it.

Unless I misunderstood you?

I did not yet entirely read your long email. I will get to that at home probably. 

But in short I guess I'm against node.op (type) being a string, this means there will be a lot of string matching. in contrary to a self.write(node.op) if the op would have its own type.

The op node would be very simple. But we can cleaner match different types. Is and equals for example. 

In my current code I did rewrite it as string. But I did not like where it is going.. I would argue that a nodetype for each operator would decouple the types from it's string representation in each language.

I will post this thread/parts of it on the mailinglist.. and I would say that we would continue there.

__________________________________________________________________________________________
__________________________________________________________________________________________

[Dexter]

Well, mapping string-to-string is trivial with a dictionary. Mapping class-to-class is a bit more tricky, in my opinion. Yes, you can do it with a dictionary, but it seems somehow less clean to do it polymorphically. Remember, that "and" and "&&" are indeed not the same, but they are both represented by the OP_AND ("&&") operator type. The specialization that turns python into javascript changes OP_AND into GETATTR("__and__") anyway, so this will not be seen by the javascript writer.

I've implemented it like this: (look at the operator_map and get_operator attributes)

https://github.com/neoel/pyjaco/blob/devel/pyjaco.new/ist/reader.py

you;re a bit right, we will get the mapping anyway. And it is better to have one operator node. But I don't quite know how to deal with all of the operators.. 

I've got in my mind something saying that not every operator has a magic method ( can anyone tell me what the magic method for 'is' is?)

Well, yes, the __fication is obvious. Typeification is another pretty simple step, where we need to turn all literals into python literals. For example, 5 becomes $c5, 42 becomes $PY.int(42) (since it does not have a pre-defined constant), "foo" becomes $PY.str("foo"), and so on.

Finally, semantification is where we change classes, methods and control structures to conform to python semantics. For example, the exception handling is vastly different, and if (foo) needs to be if (js(foo)) and so on. This is what currently represents the bulk of the compiler code.

So in other words.

__ification (__ifyTransformer) > adding all the magic methods.

typeifycation (TypifyTransformer) > explicitly make every literal a python literal. 

sematification (SemantifyTransformer) > making sure python's semantics (functions, classes, errorhandling)

Where does the name spacing fit in? determining if a variable is in a known scope, possibly wrapping the var in a lazy python object.

In your example> if (foo) wil be if (js(foo)). Doesnt it need to be if (foo.PY$__nozero__())?

Greetings.

[Tobias Baum]

hey everybody,

good to here that there is still some movement around pyjaco.

I for myself started testing the rapydscript, discussed previously.
I have to admit that i'm quit satisfied with this framework at the
moment. I can write programs
that are very close to what python is and i have full control over the DOM.

And i probably have todo only minor changes to my code to make it
compatible with next generation py2js compilers.

Can somebody with a deeper knowledge of the python to javascript
compilers explain,
what the major benefit of the new pyjaco will be compared to other
solutions such as pyjamas or rapydscript.

As I understand this issue, pyjamas is full off features, which are
not well maintained and and also not really compatible to how browsers
and the web is working. RapydScript is to me like the opposite it
takes the good thinks from JavaScript combine with standard-libs (with
future extensions in mind) and syntax from python, and has the
potential to grow over time to be more and more compatible to python,
based on the existing code-base. The integration of source-maps seems
possible, too.

I know the competition is in general a good thing, but i don't like to
reinvent the wheel.

Regards,
Tobias

[Dexter]

Hi, 

It's nice to see people with different backgrounds are listening in :).

The new pyjaco changes are mainly internal, we wanted to make the code more modular and loosely coupled, with better api's to adjust the different use cases of pyjaco.

The old code comes from an old line of forks and legacy. We wanted to clean that up, by rewriting the interals. 

Although I don't know much about rapidscript and pyjamas. My view is that pyjamas is a gwt clone to python, with a python to javascript compiler tightly coupled into the framework.

Rapidscript is more similar with pyjaco, but does not try to be python. 

Pyjaco tries to enable devs to just write python (with as many batteries as possible).

A feature i'm really looking forward to is boxing javascript variables (I want to have some sort of lazy pythonic object wrapping a javascript object). So we can seamlessly integrate javascript in our framework. (not using JSVar anymore).

And indeed, the new design should enable us to easily generate sourcemaps for debug purposes. Enabling us to create proper python tracebacks. 

We might be even able to let the chromedevtools/firebug display python/set breakpoints in python enz. and just debug python in the browser (long way to go).

Greetings, Dexter

--
You are subscribed to the Google Group pyj...@googlegroups.com
To unsubscribe from this group, send email to
pyjaco+un...@googlegroups.com

[gordon pendleton]

Dexter,

 

This is exciting.  It is good to see forward momentum for the pyjaco project.  Keep up the good work!

 

Thanks,

Gordon

[TB]

Hi, 

It's nice to see people with different backgrounds are listening in :).

 
Just consider me as an end-user that would like to have a usable/stable framework, and is willing to support one of the frameworks. Probably more in terms of testing/debugging and documenting, but i guess this are two main reasons why most open-source projects are not successful.

The new pyjaco changes are mainly internal, we wanted to make the code more modular and loosely coupled, with better api's to adjust the different use cases of pyjaco.

The old code comes from an old line of forks and legacy. We wanted to clean that up, by rewriting the interals. 

 

Although I don't know much about rapidscript and pyjamas. My view is that pyjamas is a gwt clone to python, with a python to javascript compiler tightly coupled into the framework.

Rapidscript is more similar with pyjaco, but does not try to be python. 

Pyjaco tries to enable devs to just write python (with as many batteries as possible).

I think rapydscript is trying to be python. But the core-developer accepts the fact, that there will be no complete python implementation for the browser in the near future for various reasons (see the website https://bitbucket.org/pyjeon/rapydscript) and this blog http://pyjsblog.blogspot.de/). Instead he uses an approach that is lightweight, working for end-users from the beginning and can grow over time.

Bye the way he also has the best documentation of all py2js projects and you can install it over the pypi.

A feature i'm really looking forward to is boxing javascript variables (I want to have some sort of lazy pythonic object wrapping a javascript object). So we can seamlessly integrate javascript in our framework. (not using JSVar anymore).

This already is available in rapydscript if understand you correctly. 

And indeed, the new design should enable us to easily generate sourcemaps for debug purposes. Enabling us to create proper python tracebacks. 

We might be even able to let the chromedevtools/firebug display python/set breakpoints in python enz. and just debug python in the browser (long way to go).

rapydscript does not support it either right now, but since the javascript-code it generates is very clean, you can use it to find compiler or user errors. The current rapydscript compiler checks for common user errors. And maybe sourcemaps can be integrated later, too.
 

Greetings, Dexter

Don't get me wrong in don't want to convince you to use or support rapydscript. I also do not want to stop your enthusiasm. I just want to figure out, what could make the upcoming pyjaco superior over the other approaches, which use-cases you have in mind when reimplement/develop it and when will there be a first usable version