merging py2js with mattpap.github.com/py2js

[Peter Rust]

Jonathan, Niall, et al,

 

Last night I got an email from Ondrej, who is part of the hpfem group and one of the two authors of a different py2js on github about merging the two projects (both MIT-licensed). They started about a week ago, building a new system because they were fed up with pyjamas and pyvascript. We’ve exchanged a few long emails and, from all our discussion, Ondrej and I almost identical in our approach.

 

They’ve made quite a bit of progress in the past week and are fairly similar to us in their feature-set – they have some features we don’t have and vice-versa. Their javascript library architecture mirrors ours almost exactly (wrappers for list/dict/tuple/etc), but they have tuples implemented and we have an import mechanism and args/kwargs unpacking. Their python library architecture is flatter than ours, they don’t create an in-between tree of objects, as we are doing. I am inclined to simplify ours and follow their approach (Niall, do you know of reasons not to do this or gotchas we might encounter?)

 

Ondrej has proposed using our JS library as the base JS library and migrating any extra stuff they have into it and using their Python library as the base Python library and migrating our stuff into it. This sounds like a good plan to me and I’d like to move on it – any objections?

 

We’ll probably move the repository to github, but use http://hg-git.github.com/ so that mercurial users can access the repository transparently (I have a co-worker who is doing this with one of our projects and it’s worked great for him).

 

There will be a lot of email traffic, especially over the next couple of weeks as we get everything sorted out – and we’d like to post the emails we’ve sent privately to the group, so others can see them and so they’ll be saved for historical purposes.

 

Is it ok to use the js4py group for that, or would should we create a py2js group, specifically for the project?

 

-- Peter Rust

--
You received this message because you are subscribed to the Google Groups "JavaScript for Python programmers" group.
To post to this group, send an email to js...@googlegroups.com.
To unsubscribe from this group, send email to js4py+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/js4py?hl=en-GB.

[Jonathan Fine]

Hello Peter

I'd like the two project to start merging but I don't want to rush into it.  (I think we will learn more by giving it some time.)

A good start would be to continue use both repositories (if all are willing to be flexible on the DVCS), and to give members of both projects commit rights on both projects.

I've installed the hg-git extension, which seems to be working fine for me so far, and have done another Emacs js2-mode nit-picking exercise.

I'm very much in favour of the merge process taking place in this list.

So to answer the questions you highlight:

Yes to continuing to use js4py list.

Let's think about merging the libraries.

I think the in-between tree of objects is likely to be useful in the future.



Jonathan

[Ondrej Certik]

Hi everybody,

On Apr 29, 11:56 am, "Peter Rust" <pe...@cornerstonenw.com> wrote:
> Jonathan, Niall, et al,
>

> Last night I got an email from  <http://ondrejcertik.blogspot.com/> Ondrej,
> who is part of the  <http://hpfem.org/main/> hpfem group and one of the two
> authors of a different py2js on  <http://mattpap.github.com/py2js/html/>

> github about merging the two projects (both MIT-licensed). They started
> about a week ago, building a new system because they were fed up with
> pyjamas and pyvascript. We've exchanged a few long emails and, from all our
> discussion, Ondrej and I almost identical in our approach.
>
> They've made quite a bit of progress in the past week and are fairly similar

> to us in their feature-set - they have some features we don't have and

> vice-versa. Their javascript library architecture mirrors ours almost
> exactly (wrappers for list/dict/tuple/etc), but they have tuples implemented
> and we have an import mechanism and args/kwargs unpacking. Their python
> library architecture is flatter than ours, they don't create an in-between
> tree of objects, as we are doing. I am inclined to simplify ours and follow
> their approach (Niall, do you know of reasons not to do this or gotchas we
> might encounter?)
>
> Ondrej has proposed using our JS library as the base JS library and
> migrating any extra stuff they have into it and using their Python library
> as the base Python library and migrating our stuff into it. This sounds like

> a good plan to me and I'd like to move on it - any objections?
>
> We'll probably move the repository to github, but usehttp://hg-git.github.com/so that mercurial users can access the repository

> transparently (I have a co-worker who is doing this with one of our projects
> and it's worked great for him).

I just played with hg-git and we can have the hg repository at
bitbucket and the same git repository at github, as hg can pull/push
into both. In particular we can then choose where our main official
repo will be (either bitbucket or github) and Peter can then use hg
post his devel branches to bitbucket and I can use git and post my
devel branches to github, and merging will be done using "hg pull/
merge" from both bitbucket and github, and then by "hg push" either to
bitbucket or github (depending on where our official repo is), so this
should not be a big deal.

>
> There will be a lot of email traffic, especially over the next couple of

> weeks as we get everything sorted out - and we'd like to post the emails

> we've sent privately to the group, so others can see them and so they'll be
> saved for historical purposes.
>
> Is it ok to use the js4py group for that, or would should we create a py2js
> group, specifically for the project?

Yes, let's discuss all the other points in the proper mailinglist,
either this one, or some other one.

I believe that py2js should have it's own mailinglist (as there will
be quite some traffic) and so the question really is whether we can
use js4py as the main mailinglist for py2js, and I would probably say
that we should rather create a new mailinglist just for py2js.

Ondrej

[mccniall]

Hi,

About keeping that "in-between" tree of objects.

At one point I was thinking about having completely different backends
for different output languages (actually got a couple of potential
uses for a python backend, but that might be better off as a separate
project). Having a way of isolating the frontends and backends from
each other would keep things a bit more pluggable. However, I don't
feel _that_ strongly about it.

Best
Niall

[Ondrej Certik]

Hi Jonathan,

just saw your email, let me reply:

On Thu, Apr 29, 2010 at 1:44 PM, Jonathan Fine
<jonatha...@googlemail.com> wrote:
> Hello Peter
>
> I'd like the two project to start merging but I don't want to rush into it.
> (I think we will learn more by giving it some time.)
>
> A good start would be to continue use both repositories (if all are willing
> to be flexible on the DVCS), and to give members of both projects commit
> rights on both projects.

Well, as a start, I would rebase (using "git filter branch") my
repository on top of yours, so that we can easily merge branches and
share common code that we agree on. E.g. having just one repository,
but different branches. As it is now, it's quite difficult to
pull/push some common patches.

> I've installed the hg-git extension, which seems to be working fine for me
> so far, and have done another Emacs js2-mode nit-picking exercise.
>
> I'm very much in favour of the merge process taking place in this list.
>
> So to answer the questions you highlight:
>
> Yes to continuing to use js4py list.

I am a bit confused about all the projects here. So far we have the
following repositories, I tried to write there who has push access to
it:

http://code.google.com/p/js4py/source/checkout (Jonathan, Niall)
http://bitbucket.org/PeterRust/py2js/ (Peter, Jonathan)
http://github.com/mattpap/py2js (Ondrej, Mateusz)

And we have one mailinglist for all of this: js4py.

What is the purpose of the js4py code?
The purpose of Peter's py2js and ours py2js is almost (or entirely)
identical, and I just want to merge it into one repository, in
particular I have no problem using yours as the start, and we can keep
controversial modifications in a branch, and things that we agree on
we'll have in the main repository.

> Let's think about merging the libraries.
>
> I think the in-between tree of objects is likely to be useful in the future.

We can keep both your py2js and ours py2js around and let it for the
users to choose. However, we need good tests, so that I can adapt ours
to pass your tests and vice versa.

Also:

On Thu, Apr 29, 2010 at 2:17 PM, mccniall <niall_m...@yahoo.com> wrote:
> Hi,
>
> About keeping that "in-between" tree of objects.
>
> At one point I was thinking about having completely different backends
> for different output languages (actually got a couple of potential
> uses for a python backend, but that might be better off as a separate
> project).  Having a way of isolating the frontends and backends from
> each other would keep things a bit more pluggable.  However, I don't
> feel _that_ strongly about it.

I suggest we keep both things around. For me, I just want a simple,
but robust python to JS converter, that is easy to understand and easy
to fix and maintain, so from this point of view, having a simple file
that takes python AST and emits JS directly is imho the best approach,
but I understand that to achieve your goal (e.g. having multiple
backends) it makes sense to use your approach. However, if I was
emiting a Python code (let's say) or some other code, I would just
write another AST converter from scratch. If you look at this code
here:

http://github.com/mattpap/py2js/blob/master/py2js.py

there really isn't anything to share, except the AST, which is in
Python itself (2.6 and above).

Also it's a question who wants to maintain the converter and I have no
problems maintaining our converter to be compatible with your tests
and as long as you maintain your converter too, then we just need good
tests and then we are fine.

Ondrej

[Peter Rust]

Ondrej,

 

Did you see Jonathan’s message: http://groups.google.com/group/js4py/msg/aae0f4e59ffc9058?hl=en-GB

 

I think your emails may have crossed or he may have sent it just before you joined the group. I’m inclined to continue using the js4py list for now (though in the long run we should make one named “py2js” if only to reduce the acronym confusion) and to start the merge while retaining separate repositories. Though this may be a bit more work, it’ll give us a lot more visibility as we start making the merge. The first thing I’ll do is look closely at the in-between tree and research the benefits it gives us over the standard library’s AST tree – this should help us arrive at a clear answer on which code to use as the base for the python-side.

 

As far as the Javascript side, I think we can go ahead and start treating “pylib.js” as the base (though I think we should rename it to py-builtins.js, which is a little clearer) and pull the additional features from your builtins.js into it (continuing to use bitbucket/mercurial for the time being). If, somehow, we’re unable to agree on the same python architectures (I would be very surprised, but you never know), we can continue sharing the javascript library, as our approaches there are already virtually identical.

 

-- peter

[Ondrej Certik]

On Thu, Apr 29, 2010 at 2:37 PM, Peter Rust <pe...@cornerstonenw.com> wrote:
> Ondrej,
>
>
>
> Did you see Jonathan’s message:
> http://groups.google.com/group/js4py/msg/aae0f4e59ffc9058?hl=en-GB
>
>
>
> I think your emails may have crossed or he may have sent it just before you
> joined the group. I’m inclined to continue using the js4py list for now
> (though in the long run we should make one named “py2js” if only to reduce
> the acronym confusion) and to start the merge while retaining separate

Well, if we need it in the long run, we can just as well create it now. :)

> repositories. Though this may be a bit more work, it’ll give us a lot more
> visibility as we start making the merge. The first thing I’ll do is look
> closely at the in-between tree and research the benefits it gives us over
> the standard library’s AST tree – this should help us arrive at a clear
> answer on which code to use as the base for the python-side.

Let me reply to this below.

>
>
>
> As far as the Javascript side, I think we can go ahead and start treating
> “pylib.js” as the base (though I think we should rename it to
> py-builtins.js, which is a little clearer) and pull the additional features
> from your builtins.js into it (continuing to use bitbucket/mercurial for the

Yes, that part is clear, I like your approach better, and we can just
port there some stuff from our builtins.js and we should name it

py-builtins.js

and it should contain just python builtins, e.g. I think that the
python standard library should be implemented using py2js and direct
calls to javascript, e.g. for example the math.sqrt would be
implemented by math.py:

def sqrt(x):
return Math.sqrt(x)

and so on. After importing python modules (resp. emulating it) works fine.

> time being). If, somehow, we’re unable to agree on the same python
> architectures (I would be very surprised, but you never know), we can
> continue sharing the javascript library, as our approaches there are already
> virtually identical.

Actually, I would first try to write better tests, so that I can
easily run them and try to satisfy them using our approach (I am
pretty sure it will be quite easy -- we'll we'll see) and then we have
two different approaches to the same thing and we can compare it. If
we agree to choose one, we choose it, if not, we will have both
things. So imho that's not a big deal.

From my experience the trickiest part is to write the javascript
library, that supports python like coding.

Ondrej

[Ondrej Certik]

Let me maybe elaborate on this:

I think that we all agree that we need a good support on the JS side
in a form of a good documented, tested, clearly written and standalone
JS library (that works in all browsers and JS versions).

However, I think that everyone of us might need a bit different
support from the translator. In particular, I can see the following
design decisions:

* simple script using ASTs, vs. more complex frontend vs backend approach
* generating a code that uses pure javascript, e.g. [] becomes an
Array and the result would be a python syntax to javascript
* generating a code that uses py-builtins.js, and there are several
decisions to be made as well:
1) using py-builtins lists, tuples, dicts and iterators only, e.g.
comparisons are javascript based etc.
2) emulating __eq__(), so that you can write a code like

if some_list == []:
do_something()

which means that "==" gets transformed into some_list.__eq__([]).
3) overloading "+", "-" and other things, e.g. do the Python
operators method resolution (calling the __mul__ stuff and handling
NotImplemented stuff)
4) emulating classes and especially function calls, so that things
like __call__() overloading works (this might make things really slow)
* generating different code for different JS versions, e.g. supporting
the "yield" statement in JS in ff3.


I don't have a clear opinion on the above myself, so I really don't
expect that we can agree on something together. We'll probably have to
support all things that people might need/want and are willing to
maintain.

On the other hand, as to the js-builtins.js, I think that it can
support all of the above, in particular it can implement __eq__(),
that you, as a JS user, might or might not call, it can implement
__mul__ and other things, as well as methods to do the proper Python
like method resolution, e.g. something like Mul(a, b) and it would
properly call a.__mul__(b), if it returns NotImplemented or the method
doesn't exist, it would try b.__mul__(a) and so on.

And it's then just a matter of the translator to either generate code
like "a*b" vs. "Mul(a, b)", or "a == b" vs. "a.__eq__(b)". In fact, I
think that we should have an "eq(a, b)" method, that would call
a.__eq__(b) internally if it exists, and if it doesn't, simply try to
compare things using the javascript way.

So that's roughly how I see it so far.

[Peter Rust]

Jonathan, Niall, Ondrej,

I went ahead and changed the name to py-builtins and fixed all the internal
and external references to it and pushed the changes. I agree with Ondrej
that implementing parts of the python standard library should be done in
other files (and the idea of writing the implementation in python and using
py2js to convert it to Javascript is a fun one).

> I would first try to write better tests, so that I can
> easily run them and try to satisfy them using our approach

Niall's tests are awesome, functionality based tests (I'm ashamed I didn't
pay attention to them sooner, I'm really impressed!). My
documentation-example tests are more style tests, where I wanted to showcase
on the site how clean the generated code was and run the examples to make
sure we didn't regress in this area.

I would prefer, if possible to reuse as much as possible and use flags or
even possibly different python engines to generate different kinds of
Javascript. If we can reuse the tests, the site, the documentation, the
lists, etc, the more benefit everyone will get.

Regarding the javascript library -- yes, there are many ways you could do it
and I agree that it's mostly about "who's willing to write and maintain the
feature". If someone really wanted to write the javascript functions
__eq__(), __mul__(), etc to support overloading operators, that's fine by
me, but if the javascript library starts getting large and unwieldy, I would
propose breaking it into multiple .js files that support different pieces of
functionality and writing a simple python script that puts together the
final .js file based on preference flags.

> I want to have a readable code, that's clear, but imho,
> that is just the way it is actually implemented in py2js.py.
> I don't like testing this too much, as then when you change how
> things are implemented internally, you need to change the tests as well
Agreed that you don't want to write too many overlapping "beauty" tests for
the reason you mention. I wouldn't write a bunch of these, like I would with
extensive regression tests, but rather as many as make sense for the
documentation on the website. I'll maintain these with whatever py2js engine
I'm maintaining, since readability is my big thing.

-- peter

-----Original Message-----
From: js...@googlegroups.com [mailto:js...@googlegroups.com] On Behalf Of
Ondrej Certik
Sent: Thursday, April 29, 2010 2:55 PM
To: js...@googlegroups.com
Subject: Re: merging py2js with mattpap.github.com/py2js

On Thu, Apr 29, 2010 at 2:37 PM, Peter Rust <pe...@cornerstonenw.com> wrote:
> Ondrej,
>
>
>
> Did you see Jonathan's message:
> http://groups.google.com/group/js4py/msg/aae0f4e59ffc9058?hl=en-GB
>
>
>
> I think your emails may have crossed or he may have sent it just before
you
> joined the group. I'm inclined to continue using the js4py list for now
> (though in the long run we should make one named "py2js" if only to reduce
> the acronym confusion) and to start the merge while retaining separate

Well, if we need it in the long run, we can just as well create it now. :)

> repositories. Though this may be a bit more work, it'll give us a lot more
> visibility as we start making the merge. The first thing I'll do is look
> closely at the in-between tree and research the benefits it gives us over

> the standard library's AST tree - this should help us arrive at a clear

[Ondrej Certik]

Hi,

On Thu, Apr 29, 2010 at 3:43 PM, Peter Rust <pe...@cornerstonenw.com> wrote:
> Jonathan, Niall, Ondrej,
>
> I went ahead and changed the name to py-builtins and fixed all the internal
> and external references to it and pushed the changes. I agree with Ondrej
> that implementing parts of the python standard library should be done in
> other files (and the idea of writing the implementation in python and using
> py2js to convert it to Javascript is a fun one).
>
>> I would first try to write better tests, so that I can
>> easily run them and try to satisfy them using our approach
> Niall's tests are awesome, functionality based tests (I'm ashamed I didn't
> pay attention to them sooner, I'm really impressed!). My
> documentation-example tests are more style tests, where I wanted to showcase
> on the site how clean the generated code was and run the examples to make
> sure we didn't regress in this area.
>
> I would prefer, if possible to reuse as much as possible and use flags or
> even possibly different python engines to generate different kinds of
> Javascript. If we can reuse the tests, the site, the documentation, the
> lists, etc, the more benefit everyone will get.

I am still confused about my question here:

http://groups.google.com/group/js4py/msg/f2bfa1e9a188e0e9

in particular:

-----------------

I am a bit confused about all the projects here. So far we have the
following repositories, I tried to write there who has push access to
it:
http://code.google.com/p/js4py/source/checkout (Jonathan, Niall)
http://bitbucket.org/PeterRust/py2js/ (Peter, Jonathan)
http://github.com/mattpap/py2js (Ondrej, Mateusz)
And we have one mailinglist for all of this: js4py.

-----------------

e.g. you took js4py and added more features there and put it to py2js
at bitbucket?


In any case, I tried your latest version of py2js and the
'unittest.sh' still fail. How can we fix this?

>
> Regarding the javascript library -- yes, there are many ways you could do it
> and I agree that it's mostly about "who's willing to write and maintain the
> feature". If someone really wanted to write the javascript functions
> __eq__(), __mul__(), etc to support overloading operators, that's fine by
> me, but if the javascript library starts getting large and unwieldy, I would

Yes, I have already implemented beginnings of __eq__() into our
py-builtins.js. I don't need __mul__() yet, so I'll leave this for
some future improvements.

> propose breaking it into multiple .js files that support different pieces of
> functionality and writing a simple python script that puts together the
> final .js file based on preference flags.

How do other JS libraries solve this problem? I think that we should
eventually split it into multiple files.

>
>> I want to have a readable code, that's clear, but imho,
>> that is just the way it is actually implemented in py2js.py.
>> I don't like testing this too much, as then when you change how
>> things are implemented internally, you need to change the tests as well
> Agreed that you don't want to write too many overlapping "beauty" tests for
> the reason you mention. I wouldn't write a bunch of these, like I would with
> extensive regression tests, but rather as many as make sense for the
> documentation on the website. I'll maintain these with whatever py2js engine
> I'm maintaining, since readability is my big thing.

I agree it's cool to have a few that would be used as a show case
example, so that we are sure it still looks nice. +1 to that.

[Peter Rust]

Ondrej,

Not much time here, so I'll just give a super-quick reply to help answer
your question. The "js4py" caused me a lot of confusion too -- as well as a
local pythonista Nick, when I was introducing him to the libraries.

Js4py is just a loose collection of resources that Jonathan put together of
various utilities, slides and notes that would help Python programmers who
were learning about Javascript or starting to get into Javascript
development. It includes a talk that Jonathan gave at a European Pycon and
some really good notes on differences between the two languages and Niall's
original py2js as well as a partially-done revision that Jonathan started
called py2js-v02.

Jonathan encouraged me to take the py2js from googlecode (which was
essentially the same as the one on Niall's site) and bring it into
bitbucket, since he was moving toward bitbucket himself, and start active
development on it again.

-- peter

-----Original Message-----
From: js...@googlegroups.com [mailto:js...@googlegroups.com] On Behalf Of
Ondrej Certik
Sent: Thursday, April 29, 2010 4:01 PM
To: js...@googlegroups.com
Subject: Re: merging py2js with mattpap.github.com/py2js

[Ondrej Certik]

Hi Peter!

On Thu, Apr 29, 2010 at 4:28 PM, Peter Rust <pe...@cornerstonenw.com> wrote:
> Ondrej,
>
> Not much time here, so I'll just give a super-quick reply to help answer
> your question. The "js4py" caused me a lot of confusion too -- as well as a
> local pythonista Nick, when I was introducing him to the libraries.
>
> Js4py is just a loose collection of resources that Jonathan put together of
> various utilities, slides and notes that would help Python programmers who
> were learning about Javascript or starting to get into Javascript
> development. It includes a talk that Jonathan gave at a European Pycon and
> some really good notes on differences between the two languages and Niall's
> original py2js as well as a partially-done revision that Jonathan started
> called py2js-v02.
>
> Jonathan encouraged me to take the py2js from googlecode (which was
> essentially the same as the one on Niall's site) and bring it into
> bitbucket, since he was moving toward bitbucket himself, and start active
> development on it again.

Ah, I understand it finally. Thanks!

[Jonathan Fine]

Hell Peter, Ondrej and others,

Here are some suggestions for doing the merge.

0. Give members of each project commit rights on the other.

1.  Give a name to each of the py2js projects.  I'm going to use pear and plum here.

2. In the pear project move everything into a new folder <root>/pear, and the same for the plum project.

3. Chose a shared repository, either on BitBucket or GitHub.

4. Add all of pear (including history if possible) to plum project (or vice versa).

5. Create a folder <root>/test_py2js that contains test scripts and data.

6. Create a script <root>/bin/run_tests.py that takes 'pear' or 'plum' as parameter and which will run the tests.

I'm not so sure about (5) and (6). 

For (3) my preference is to use BitBucket.  One reason is that I like better the support it provides for issues.  Compare
    http://bitbucket.org/jfine/python-testutil/issue/26/allow-runtestspy-to-discover-modules-that-need
    http://github.com/mattpap/py2js/issues#issue/4

--
Jonathan