proposed py2js changes

[Peter Rust]

Niall,

 

I’m interested in making some significant adjustments to py2js. I had actually started writing the same thing last weekend using Python 2.6’s “ast” module (as opposed to 2.5’s “_ast” module). It is a relief and an encouragement to see all the code you’ve written & open-sourced.

 

My goal is a system that’s less ambitious than the pyjamas translator – just converting syntax and providing some of Python’s syntactic sugar without trying to smooth over all the differences between the two languages.

 

The #1 most important thing for me is generating clean code. This is critical both for debuggability and also for publishing the generated Javascript publicly (I would like to create cross-language modules that can be consumed both by Python users and Javascript users).

 

When I started using py2js, I found a number of things I would like to work on. One of them I would like to consider is porting it to the Python 2.6 “ast” module, as it is higher-level than the “_ast” module.

 

Another is allowing import of pure javascript namespaces that aren’t written in Python and aren’t being converted with py4js. For instance, webtoolkit’s base64 javascript module could be imported with python like “from webtoolkit import base64″, which might be translated to javascript like “import(‘base64′).from(‘webtoolkit’)”.

 

There are a number of smaller changes, things like putting variable declaration and assignment on the same line (“var n = …”), avoiding repeated, unnecessary variable declarations (if a variable is assigned in an “if” scope, a second, unnecessary “var” statement is generated), changing “new Kwarg” to something more compact, perhaps simply “kw”, removing some unnecessary grouping parenthesis and String() typing around string concatenation, and possibly looking at a non-generator approach to “for” loops.

 

I would also want to do lots of documentation/examples/doctests (using Sphinx) to really widen the web presence. I think a project like this could get a lot of steam if people saw the quality of the javascript being generated and that it really is a viable alternative to writing client-side code in JS (I would want it to operate cleanly with Prototype, jQuery and other JS frameworks).

 

I would also like to write a little helper utility that uses a FileSystemWatcher to auto-convert any .py file in a set of folders to a sibling .js file any time it is saved, to make the development save/refresh cycle faster.

 

Please let me know if you would like me to (a) move the py2js source from google code to my bitbucket account and start developing there (my preference), (b) contribute to the code in the google code repository, (c) fork the code and call my branch something else, with you pulling just the pieces (if any) that you want in the official branch, or (d) something else.

 

-- Peter Rust

Developer, Cornerstone Systems

--
You received this message because you are subscribed to the Google Groups "JavaScript for Python programmers" group.
To post to this group, send an email to js...@googlegroups.com.
To unsubscribe from this group, send email to js4py+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/js4py?hl=en-GB.

[Jonathan Fine]

I welcome Peter's interest in Niall's work, and also his suggestion that we move development to Mercurial and BitBucket (which I am now using: http://bitbucket.org/jfine/).

--
Jonathan

[mccniall]

Hi Peter,

Great to hear you are interested in taking things further with py2js -
I do hope you find it useful - I've certainly got no objections to
your moving the code to bitbucket and working on it from there if that
is more convenient. I'd certainly like to hear of any improvements
and fixes you make, or general observations.

I think the changes and directions you outline make a lot of sense.
Being able to generate clean javascript which would be easy to debug
and understand is an important goal to adopt (I don't think py2js is
too far off but it certainly wasn't my top priority) and not one that
similar projects always have.

My feeling is that if you wanted to make larger changes (for example,
switching the generated javascript to use prototypal inheritance which
would arguably enhance readability) it might be tricky to work work
with the code in its current rather unwieldy form. Some refactoring
(or rewriting) might be needed. Smaller changes should be feasible.
Removing functionality that isn't worth the complexity (multiple
inheritance support, perhaps) may or may not make sense.

In general, I think the approach of keeping the generated code as
simple as possible, exploiting similarities between the languages, and
documenting the situations where this approach doesn't work makes a
lot of sense for some potential usages. Its a syntax driven approach
rather than one that tries to get deep into the semantics.

Best,
Niall


On Apr 22, 1:10 pm, Jonathan Fine <jonathan.fi...@googlemail.com>
wrote:

[Peter Rust]

Niall,

Thanks for the green light! I'm planning on pouring a good deal of time into
py2js this weekend -- hopefully I'll have a working iteration on the project
posted to bitbucket by Monday morning.

Primarily, I will focus on cleaning up the generated JS and writing lots of
documentation and examples and designing a snazzy logo. I might also
experiment a bit with converting to the Javascript-AST approach Jonathan
began with "py2js-v02" and/or converting to Python 2.6's "ast" module.

Regarding inheritance, my instinct at this point is to provide a variety of
pluggable options, since there is so much divergence on this point in the
Javascript community. I’m most familiar with using the classical inheritance
system in code.google.com/p/base2 and prototypejs.org, but I've used a
couple variations of prototypical inheritance outlined in the "Details of
the Object Model" section of Mozilla's Javascript 1.5 Guide
(https://developer.mozilla.org/en/Core_JavaScript_1.5_Guide) -- and Douglas
Crockford and Google Closure both have their own approaches, too (compared
here: http://bolinfest.com/javascript/inheritance.php).

I doubt I'll get to it this weekend, but I would love making py2js generate
base2, Prototype, Crockford, Google Closure or one of the two Mozilla styles
of inheritance, all based on how it's configured. Not all of them map well
to Python's inheritance model (some will throw an error when multiple
inheritance is encountered in the python source), but client-side developers
will be able to generate a style of Javascript that they're familiar with
and that is consistent with the rest of their project and/or team.

Similarly I would like to see the generated whitespace be extremely
configurable (I would stick with PEP 8 where possible for the defaults, but
some of them I like to deviate from, especially for Javascript).

> keeping the generated code as simple as possible,
> exploiting similarities between the languages, and
> documenting the situations where this approach doesn't work

Yes, well summed-up. In some situations -- "for...in" loops, for example --
I can imagine using some simple detection techniques to see if we know for
sure that the variable being iterated is indeed a list/array and in this
case rendering a vanilla for loop with an index, which is faster and
cleaner. If the variable isn't a list or we can't figure it out easily (very
likely), we can fall back on a more robust generator approach which supports
a StopIteration exception, etc.

Supporting identifiers that begin with "$" will be trickier, but very
necessary to support transparent interoperability with jQuery and Prototype
(something I rely on heavily for client-side development). At this point I'm
thinking of a configurable "S_" convention, which, when found as the prefix
for a python identifier, would be translated into "$" in the destination
Javascript.

-- Peter Rust
Developer, Cornerstone Systems

[Jonathan Fine]

Here's something that's come to my mind recently.  It's about how a.b and a['b'] are identical in JavaScript and different in Python (of course).  Also, in JavaScript hashes don't start empty.  They inherit attributes such as toString.

Python has setattr and gettattr.  My idea is to provide semantically similar JavaScript functions, except that they prefix the key string with a character, say '#'.

One could then map (perhaps only for objects of a particular class) Python x = a[b] to JavaScript x = getattr(b).  One might also want a JavaScript getkeys() function.

[Peter Rust]

Jonathan,

 

It’s encouraging to see your continued interest and that you’re chewing on these problems. I was thinking about this one myself early this morning – the difference between python dicts and javascript objects and the distinction python makes between dot notation and bracket notation, as opposed to Javascript treating them the same.

 

For a little background, the Prototype Javascript library (a Ruby-influenced lib) created a Hash object which, at first, decorated a standard JS object with helpful methods (.keys(), .values(), .each(), .clone(), .toQueryString(), etc). You would instantiate it via “new Hash({ 'my_key': 1 })” or via “$H({ 'my_key': 1 })” and it simply added functionality to the native Javascript object – you could retrieve properties via dot notation or bracket notation of the Hash function .get(). However, in later versions of the library (and the current version), they made a breaking change where the Hash object stored all the data hidden away in an internal object and the only way to set or get it was via .set() and .get(). They had to do this because of naming collisions between the data stored in the Hash and the functions decorating the hash.

 

As a Javascript developer, there are rare cases when I don’t know the keys that will be used and there’s a chance of collision with Hash methods – in this case I’ll use Prototype’s Hash object. But the vast majority of the time, I’m dealing with a distinct set of keys that I know in advance and I use native Javascript objects for convenience (my preference is for dot notation, with square backet notation being a distant second and get/set methods last).

 

Given that python is more strict in this case (allowing only square-bracket notation for dicts) and Javascript is more lenient (also allowing dot-notation) – and since we’re going in the PY -> JS direction, it would work to just retain the square-bracket notation in the emitted Javascript, with the exception of naming collisions, as you’ve mentioned. Not only is there a potential for

 

I’m thinking that the best thing to do would be to give users a choice, in their project-level configuration file, whether to convert bracket notation to get/set or getattr/setattr, or whether to leave them as bracket notation:

 

1. Keep as Bracket Notation:

·         Cleaner Syntax (possibly we could allow dot notation as a variation)

·         Faster

·         With some custom JS code, can support custom getters/setters on modern browsers

·         Potential for collisions if you don’t know the keys in advance

 

2. Convert to getattr()/setattr():

·         Safer (no potential for collisions)

·         Easy, native support for custom getters/setters

·         Slightly slower

·         Slightly addition syntactic cruft (possibly we could allow get()/set() as a variation)

 

I’m not sure which would be the default. #1 generates cleaner code and perhaps would be recommended for short & simple scripts, but #2 is more robust and would be recommended for longer scripts or more complex frameworks, especially when porting Python code which distinguishes between attributes and properties (like ElementTree). Perhaps there would even be two basic configuration sets – a more “robust mode” that is safe and is very extensible and a potentially-unsafe “super-clean” mode, where the developer would need to be more aware of the differences between the languages and know the pitfalls.

 

I’d love to hear additional thoughts and feedback, it’s never good to design in a vacuum…

 

-- peter

 

From: js...@googlegroups.com [mailto:js...@googlegroups.com] On Behalf Of Jonathan Fine
Sent: Thursday, April 22, 2010 11:32 PM
To: js...@googlegroups.com
Subject: Re: proposed py2js changes

 

Here's something that's come to my mind recently.  It's about how a.b and a['b'] are identical in JavaScript and different in Python (of course).  Also, in JavaScript hashes don't start empty.  They inherit attributes such as toString.

[Peter Rust]

Jonathan,

 

Forgot to respond to this implementation idea:

 

> they prefix the key string with a character, say '#'

Not a bad idea – either prefix with a character that’s not valid for dot-notation or hide them away in an inner object.

 

-- peter

 

From: js...@googlegroups.com [mailto:js...@googlegroups.com] On Behalf Of Jonathan Fine

Sent: Thursday, April 22, 2010 11:32 PM
To: js...@googlegroups.com

Subject: Re: proposed py2js changes

 

Here's something that's come to my mind recently.  It's about how a.b and a['b'] are identical in JavaScript and different in Python (of course).  Also, in JavaScript hashes don't start empty.  They inherit attributes such as toString.

[Jonathan Fine]

Peter

I think some examples of how these things are used in real-world software would be quite helpful. I'm quite interested right now in library code, and a framework for library code.

I have some 'Miller Columns' software in development which would make a good use case.

[Peter Rust]

Jonathan,

 

Yes, I am anxious to try these ideas in real-world software and see how (or if) they fly – I’m sure things will evolve quite a bit once the rubber meets the road. We’ll start at “0.1” (or is it 0.2… or 0.3?) and let things vary wildly as we’re figuring out what works and what doesn’t.

 

I found a couple existing “pythonic” Javascript libraries:

·         http://code.google.com/p/protopy/

·         http://pyjs.info/

·         http://www.mochikit.com/

 

Google’s Closure Library also has some pythonic things, including modules and iteration (a next() method and a StopIteration exception). I haven’t looked too deeply yet, but we should look carefully at what these offer to avoid spending our time re-doing things that have already been done. Plus, we may be able to learn from others’ experiences. We may also find some people who are interested in helping with this project.

 

We don’t need to create a full client-side library (dealing with ajax/dom/etc) – users can easily import any of the existing libs for that, but we do need to cover the basics: Python’s built-in list and dict functionality as well as some of the string functionality and the basic Python built-in functions. Niall has already done some of this and for starters we only need to focus on implementing the 20% that is used 80% of the time and clearly documenting what is (and isn’t) implemented. Some things that are already built into Javascript require only slight modification (like wrapping Javascript’s built-in array/string slice() function with a custom function that can convert negative indexes based on the length of the array/string).

 

Speaking of Miller Columns, I have a set of helper functions that help with grouping data and generating tables – they’re currently written in Javascript, but I intend to port them to Python and use py2js to make them available in both Python and Javascript. They will also give it a good test-run. Also, I have a public-facing app, Project Recon, that I’d love to convert someday as well (I’m almost done converting it from IE+.Net to Firefox/XULRunner… but it’s still mostly Javascript).

 

-- peter

 

From: js...@googlegroups.com [mailto:js...@googlegroups.com] On Behalf Of Jonathan Fine
Sent: Friday, April 23, 2010 11:33 AM
To: js...@googlegroups.com
Subject: Re: proposed py2js changes

 

Peter

[Jonathan Fine]

Thank you, Peter, for your comments.

I think one of the big challenges is to get something that is widely used.  Python has a standard library, which is very widely used. JavaScript does not have a standard library.  It does not even have a module structure.  Hence my work on Jsmod.  There are people in CommonJS doing something a bit similar, but their motivation is server-side JavaScript.

[mccniall]

Using one of those pythonic JS libraries (or creating such a layer
using the same ideas) should simplify the generated code a lot (and
take a lot of complexity out of the generator). Really key idea!

BTW if you need to relicense any of the existing py2js code to use an
open source license which is compatible with another library I'm sure
that would be perfectly fine. I have a copy of the py2js code checked
into sourceforge as a component in another project I'm currently
working on, dual licensed under Apache V2.0 and GPL (I made one or two
minor changes to make it work with python 2.6, otherwise its exactly
identical to the code you've been looking at).

Cheers
Niall

On Apr 23, 8:34 pm, "Peter Rust" <pe...@cornerstonenw.com> wrote:
> Jonathan,
>
> Yes, I am anxious to try these ideas in real-world software and see how (or

> if) they fly - I'm sure things will evolve quite a bit once the rubber meets
> the road. We'll start at "0.1" (or is it 0.2. or 0.3?) and let things vary

> wildly as we're figuring out what works and what doesn't.
>
> I found a couple existing "pythonic" Javascript libraries:

> .        http://code.google.com/p/protopy/
> .        http://pyjs.info/
> .        http://www.mochikit.com/
>
> Google <http://closure-library.googlecode.com/svn/docs/index.html> 's

> Closure Library also has some pythonic things, including modules and
> iteration (a next() method and a StopIteration exception). I haven't looked
> too deeply yet, but we should look carefully at what these offer to avoid
> spending our time re-doing things that have already been done. Plus, we may
> be able to learn from others' experiences. We may also find some people who
> are interested in helping with this project.
>
> We don't need to create a full client-side library (dealing with

> ajax/dom/etc) - users can easily import any of the existing libs for that,

> but we do need to cover the basics: Python's built-in list

> <http://docs.python.org/library/stdtypes.html#sequence-types-str-unico...
> t-tuple-buffer-xrange>  and
> <http://docs.python.org/library/stdtypes.html#mapping-types-dict> dict

> functionality as well as some of the string

> <http://docs.python.org/library/string.html>  functionality and the basic
> Python built-in <http://docs.python.org/library/functions.html>  functions.

> Niall has already done some of this and for starters we only need to focus
> on implementing the 20% that is used 80% of the time and clearly documenting
> what is (and isn't) implemented. Some things that are already built into
> Javascript require only slight modification (like wrapping Javascript's
> built-in array/string slice() function with a custom function that can
> convert negative indexes based on the length of the array/string).
>

> Speaking of Miller Columns <http://en.wikipedia.org/wiki/Miller_Columns> , I

> have a set of helper functions that help with grouping data and generating

> tables - they're currently written in Javascript, but I intend to port them

> to Python and use py2js to make them available in both Python and
> Javascript. They will also give it a good test-run. Also, I have a

> public-facing app, Project Recon <http://www.projectrecon.net/> , that I'd

> love to convert someday as well (I'm almost done converting it from IE+.Net

> to Firefox/XULRunner. but it's still mostly Javascript).

>
> -- peter
>
> From: js...@googlegroups.com [mailto:js...@googlegroups.com] On Behalf Of
> Jonathan Fine
> Sent: Friday, April 23, 2010 11:33 AM
> To: js...@googlegroups.com
> Subject: Re: proposed py2js changes
>
> Peter
>
> I think some examples of how these things are used in real-world software
> would be quite helpful. I'm quite interested right now in library code, and
> a framework for library code.
>
> I have some 'Miller Columns' software in development which would make a good
> use case.
>
> --
> Jonathan
> --
> You received this message because you are subscribed to the Google Groups
> "JavaScript for Python programmers" group.
> To post to this group, send an email to js...@googlegroups.com.
> To unsubscribe from this group, send email to
> js4py+un...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/js4py?hl=en-GB.

>
> --
> You received this message because you are subscribed to the Google Groups "JavaScript for Python programmers" group.
> To post to this group, send an email to js...@googlegroups.com.
> To unsubscribe from this group, send email to js4py+un...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/js4py?hl=en-GB.

[Peter Rust]

Niall,

> Using one of those pythonic JS libraries

> or creating such a layer using the same ideas

> should simplify the generated code

Yes, this will make a big difference. The globals in the pythonic lib could
be imported into each emitted file using a module importer, like Jonathan's
jsmod, so that the lib isn't cluttering Javascript's global namespace.
However, if the lib changes Array.prototype or Object.prototype, it'll be
difficult to mask those changes from other (non py2js-generated) modules.

> I made one or two minor changes to make it work with python 2.6,
> otherwise its exactly identical to the code you've been looking at

Any chance you could send me the changes somehow? I'll be working with it
under Python 2.6 this weekend -- I'd like to avoid redoing things you've
already done...

-- peter

-----Original Message-----
From: js...@googlegroups.com [mailto:js...@googlegroups.com] On Behalf Of

mccniall
Sent: Friday, April 23, 2010 2:15 PM
To: JavaScript for Python programmers

[Jonathan Fine]

Hi

Here are two things I avoid in JavaScript programming as much as I can:

1.  Object.prototype.something = value

2.  global.something = value (where global is the global object).

(1) can always be avoided.  (2) should be used as little as possible.


Jonathan

On Fri, Apr 23, 2010 at 11:45 PM, Peter Rust <pe...@cornerstonenw.com> wrote:

Niall,

> Using one of those pythonic JS libraries
> or creating such a layer using the same ideas
> should simplify the generated code

Yes, this will make a big difference. The globals in the pythonic lib could
be imported into each emitted file using a module importer, like Jonathan's
jsmod, so that the lib isn't cluttering Javascript's global namespace.
However, if the lib changes Array.prototype or Object.prototype, it'll be
difficult to mask those changes from other (non py2js-generated) modules.

> I made one or two minor changes to make it work with python 2.6,
> otherwise its exactly identical to the code you've been looking at

Any chance you could send me the changes somehow? I'll be working with it
under Python 2.6 this weekend -- I'd like to avoid redoing things you've
already done...

--

[Peter Rust]

Jonathan,

 

Yes, (1) modifying Object.prototype is almost always a bad idea, as everything inherits from Object. This pretty much forces us down the path of a dict wrapper, as opposed to being able to use “{ … }” notation for dicts. However, I think I would modify Array.prototype (as prototypejs.org does), so that we can use “[ … ]” notation. We could have a flag to avoid this (some JS developers don’t like to modify *any* built-in prototypes) – if this flag is set, arrays would be created with “list( … )” instead of “[ … ]”.

 

As for (2), I do strongly agree with you that there should just be one global – say JSMOD, or some other object or function that does importing. Then all other modules should be hidden inside of that, as you do with your internal “module” object that’s hidden via a closure. However, when designing individual modules, I don’t usually use closures to hide properties. I like python’s approach of using single and double-underscores for things that are private or really private and leaving it at that. But I’m ok with other developers using closures to hide private things and I could do so as well if that’s the consensus for the libraries we’re working on together. The only thing I would push for is that we make it possible, within a module, to access sub-modules and functions via dot-notation. That way you have the option of doing a single import of the top-level package and then can use dot-notation to drill down as far as you need to.

 

Let me know what you think,

 

-- peter

 

From: js...@googlegroups.com [mailto:js...@googlegroups.com] On Behalf Of Jonathan Fine
Sent: Saturday, April 24, 2010 10:28 AM
To: js...@googlegroups.com
Subject: Re: proposed py2js changes

 

Hi

[Jonathan Fine]

Peter

I'll just speak on the question of imports.  You wrote:

The only thing I would push for is that we make it possible, within a module, to access sub-modules and functions via dot-notation. That way you have the option of doing a single import of the top-level package and then can use dot-notation to drill down as far as you need to.

It's common, for good reasons, to optimize JavaScript when it is deployed. One thing that can be done is to abbreviate non-local variable names.  This means the parameters to a function and variables declared in a function body.

Modules written the JSMOD way admit an additional optimisation, namely combining several JSMOD bodies into one.  But there is a bad practice that can defeat this.  Suppose I write

JSMOD({
    imports: ['wibble']
}, function(m, __){
    eval(__);

  // Do things with wibble.
   wibble = wobble;
 
  // Do more things with wibble.

));

Such code will interfere with other users of module wibble when module bodies are combined, as might
   wibble.wobble = 5;

There are two goals, (1) supporting good programming practice and (2) supporting optimisation.  Monkey-patching of modules, as above, is inconsistent with both (1) and (2).

There's a lot to be said for being able to import os and then read os.platform or os.path and so forth.  Of course, os.platform will be optimised to, say os__platform and thence to a short identifier. This is probably worth having.

But I'm not sure about packages.  I think Python and JavaScript packages might be different.  But I don't know this for sure.

I recently committed to jsmod on bb a little bit of code on module metadata.  My next task is to write Python code that will build a load list from such module metadata.

--
Jonathan

[mccniall]

> > I made one or two minor changes to make it work with python 2.6,
> > otherwise its exactly identical to the code you've been looking at
>
> Any chance you could send me the changes somehow? I'll be working with it
> under Python 2.6 this weekend -- I'd like to avoid redoing things you've
> already done...

Hi Peter,

I think the only significant change I can remember is to
pyfrontend.py. (The py2js files in my other project are located under
http://velocikit.svn.sourceforge.net/viewvc/velocikit/trunk/src/tools/py2js/)

add a new callback for ExceptHandler (seems to be new in the python
2.6 _ast):

# python 2.6
def ExceptHandler(self, ast):
return self.excepthandler(ast)

adapt to change to the way decorators seem to be handled in the python
2.6 _ast

def FunctionDef(self, ast):
decorators = set()

if 'decorator_list' in ast.__dict__:
# python 2.6
for decorator in ast.decorator_list:
decorators.add(decorator.id)
else:
for decorator in ast.decorators:
decorators.add(decorator.id)

<snip>

Cheers
Niall