My first pass on py2js
Peter Rust
Niall, Jonathan, and others,
I’ve completed my first pass on py2js. I posted the code on bitbucket, designed a logo and spent a lot of time developing the home page, which is a kind of manifesto for the project: http://bitbucket.org/PeterRust/py2js/wiki.
The vision for the project is to generate clean, readable code that more than merely “resembles” the python source – rather, it maps it one-to-one wherever possible due to the help of a pythonic javascript library, pylib.js. It also will map extremely close to the python in the generated *-ff3.js – a new concept which will make developing with py2js much more pleasant (though it’s not yet implemented – check out “Developing with Firefox 3+” section on the wiki).
I did a number of small “cleanup” items to get the wiki examples to pass (every example on the wiki is being tested – I have most, but not all, of them passing):
· “var” statements are now in-line with the first assignment of a variable
· Single-statement “if” statements don’t have curly braces
· I modified whitespace in quite a few places to follow PEP 8 as closely as possible
· I made “$” round-trip to “__S__” and back to support interoperability with popular JS libs
I also did a major refactoring, emboldened by Niall’s support for moving the generated JS into a separate pythonic js library. “jslib.py” is now “pylib.js”. It is now wrapped in a closure, so all the functions aren’t cluttering the global namespace.
You’ll notice an import mechanism at the top of pylib.js – it’s similar in concept to Jonathan’s JSMOD, but uses different syntax to more closely mirror Python. There are two globals, $import and $from, and they map very closely to python’s syntax. For single imports, you can do “var pylib = $import.pylib;”. In FF3, the proposed syntax for multiple imports is “var [pylib, webtoolkit] = $import(‘pylib’, ‘webtoolkit’);”. However, for other browsers and for the “*” case, you have to use eval, as JSMOD does: “eval($from(‘pylib’).$import(‘*’));” I searched desperately for a way to inject variables into the scope-chain without eval, but it’s impossible. I still need to verify that eval works the way I think it does across browsers. Jonathan, I know you’ve done a lot of work in this area and I don’t want to step on your toes – but I did have an idea of a different syntax and wanted to try it out. I’m more than willing for py2js to optionally generate jsmod syntax or to look at the differences and see if we can’t merge together the strengths of both systems.
Now for the humbling part. Jonathan, you were right about modifying global prototypes. I was wrong. I spent some time this weekend with a new friend, Nick Fitzgerald, who also has an interest in the project. He pushed rather hard (as you did) on the topic of modifying the built-in prototypes of Array, String and Function. The combined pressure forced me to wrestle hard with the issue. This morning I read What’s wrong with extending the DOM (by one of the leaders of Prototype – Prototype will no longer be doing this in v2.0) as well as an article he linked to: Don’t modify objects you don’t own. I can’t argue with the logic here.
There are two alternatives to modifying built-in prototypes: (1) convert object-oriented calls to functional-style calls (“my_dict.keys()” becomes “pylib.keys(my_dict)”) and (2) create a wrapper class and wrap all literals and all instances being passed in from 3rd-party Javascript (“my_dict = {}” becomes “var my_dict = dict({})”).
I lean towards option (2) because it maps more closely to the original python and because py2js has no way of knowing which object-oriented calls to turn into functional-style calls. Because of dynamic types, py2js has no way of knowing which instances are dicts. To try to trace all the code and figure it out would be a herculean effort and, ultimately, would only cover 90% of the cases. “Ah”, you say, “why don’t we just convert all method calls to popular dict functions, like “keys()” into functional-style calls?” The problem with this approach is that strings alone have 32 methods – and that’s not counting all the methods of dicts and lists. If we convert object-oriented calls using any of these method names to functional-style calls (“pylib.xxx(obj)”), it would effectively mean that custom classes are unable to name their methods any of these names because the translator would replace calls to their method with a pylib function call.
I believe the wrapping approach can take us further, especially if we’re able to make use of the prototype chain by doing something like Crockford’s beget (though not off the global Object.prototype!) to get all the built-in functionality for free.
However, there is a caveat with the wrapping approach. While py2js can auto-wrap all literals, it has no way of knowing which 3rd-party JS libraries are returning dicts, lists and strings. For instance, py2js code that calls jQuery like this “$('#status').html().lstrip()” would fail because the vanilla String that html() returns isn’t being wrapped with our str() wrapper, which provides the lstrip method. For these cases we would need to require developers to wrap all instances coming from the DOM or from 3rd party libraries in order to use python’s string methods. So the above code would need to be written as “str($('#status').html()).lstrip()”.
This might get tedious or annoying if developers are constantly using these 3rd party libraries and having to remember to wrap everywhere. If this gets to be a problem, we could provide a place for web developers to submit library wrappers for transparent interoperability. For instance, a wrapper for jQuery would call down to the real jQuery lib to do the work and then pre-wrap all results with the appropriate pylib.js types (dict, list, str, etc). For someone to use this, they would simply set the wrapper to the “$” function in the local scope, thus taking precedence over the real jQuery and – in their scope – the wrapper would be used instead. Other non-py2js-generated code would be in a different scope and would still be able to access the real jQuery directly.
['attr'] vs “.attr”
Thanks Jonathan for getting the wheels turning about the [] vs “.” issue in Javascript. I’ve been wrestling with this today as well. I think you’re right – we can’t simply convert “[]” attribute access straight to Javascript, it will wreak havoc with dict methods. We need to map it to something like get()/set() or getattr()/setattr(). For readability, I would prefer the former (shorter) syntax but we could have a flag to use the latter instead. Rather than prefixing the attribute names, I think we can set them directly on an inner object. This will be a little more performant, a little cleaner and could pave the way for the shortcut/convenience method described in the next paragraph.
The biggest annoyance of conversion from [] to get/set is that py2js has no way of knowing when [] is being used on a dict, on a list or on a tuple. Rather than converting [] for everything across the board (“my_tuple[0]” would become “my_tuple.get(0)” – yuck!), I would like to throw in a shortcut/convenience: if the attribute name is a literal number, py2js would retain the [] notation. I think that this would still drill down to the inner object to get the attribute if we used something like Crockford’s beget and have the inner object in the prototype chain (but I haven’t tried it yet).
Thanks again, Niall, for providing such a solid base of code that already works (I especially like jsmap.py, it’s super-concise!). I realize that the changes likely broke a number of existing tests. I plan to move these tests to doctest-style tests because I want a lot of example-rich documentation – as I do, I’ll update them for the changes to the structure/API so that they pass.
Please push back on anything that doesn’t sound right or anything you don’t like. You’ve already influenced my thinking significantly, proving that working as a team is better than working solo. Though it requires some painful compromises at times, the end product is better.
-- Peter Rust
You received this message because you are subscribed to the Google Groups "JavaScript for Python programmers" group.
To post to this group, send an email to js...@googlegroups.com.
To unsubscribe from this group, send email to js4py+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/js4py?hl=en-GB.
Jonathan Fine
Jonathan
Peter Rust
Thanks, Jonathan!
Also, I looked into Crockford’s beget and it turns out to be a non-performant, hacky trick that will work ok for getting, but not setting. For instance, if you wrap a JS Object { test: 'test' } with my_dict (which has all your dict methods) that sets the inner object on its prototype chain, then my_dict['test'] will get the value from the inner object (b/c it doesn’t find it on the my_dict instance itself, javascript walks up the prototype chain and keeps looking). However, if you set a value, my_dict['test2'] = 'test2', this will be set on the wrapper object, not the inner object. So if you implement my_dict.keys() to iterate the inner object, the new value won’t be there.
Plus it’s non-performant because it won’t work with normal classes – which use the prototype chain -- so you have to loop through and set each method on each new instance in the constructor.
Anyways, I’ve ditched the beget idea and am using simple wrappers… looks like we’ll be using .get() and .set() everywhere, even for lists and tuples being indexed with number literals.
Last night I implemented a bunch of dict methods (though not all of them yet) and transitioned the two list functions (extend and sort3) from being functional-style to being methods of a new list() wrapper class and added an append() method. I also renamed “sort3” to just “sort” and added a couple of internal “=== undefined” argument checks to decide what functionality to use as opposed to putting the # of arguments in the function name.
There’s really only one significant research piece remaining, that I can think of at the moment – that is whether we can reliably get cross-browser getters/setters functioning. I’m pretty sure we can get it for IE8 and all the modern browsers… it’d be a bummer to not support IE7, but in six months to a year it probably won’t matter that much.
The reason I’d like getters/setters is to implement the .__len__ property, which would need to check the .length property of the wrapped object. This will be used by the built-in len() function, in the same way the python len() function works. If it’s not possible to have getters in IE7 and we want to support the browser, we could implement len() to look for a __len__ method instead of a property. However, there are probably other pythonic internal properties like __len__, so it’d be nice to support them.
Also, I made you an administrator of the repository on bitbucket, so you have full permissions now.
-- peter
Jonathan Fine
Thank you for adding me to the project. I've looked over the code, and committed some notes and some nit-picking corrections.
I think I know how to run the tests, but not how to interpret the results. I didn't get the browser test to work. Instead I got (in Firefox)
Access to restricted URI denied" code: "1012
It's good to see energy and progress, so I hope you don't find my remarks negative.
--
Jonathan Fine
The object and inheritance models in JavaScript and Python are very different, which causes a tension for any Python to JavaScript translator. My preferred solution is to introduce into Python objects that have JavaScript properties.
Here's a related question: functions with default parameter values. In Python f(**dict(a=1, b=2)) quickly assigns values to a and b in the body of f. There's nothing like it in JavaScript, although a slow emulation can be made.
The best that can be done in JavaScript is, I think, something like
===================================================
defaults = {
'a': 123,
'b': 456,
'c': 'and so on'
}
var f = function(a, b, c) {
// Body goes here.
}
args = create(defaults); // Prototype inheritance.
args.b = 100; // Non default value.
f(args.a, args.b, args.c); // Same as f(123, 100, 'and so on')
===================================================
The equivalent Python is then something like
===================================================
def f(a=123, b=456, c='and so on'):
# Body goes here.
f(b=100)
===================================================
By the way, 'defaults' could be made an attribute of the function f.
So my general philosophy is adapt the Python semantics that it fits in nicely with JavaScript, rather than aim for 100% compatibility (with a complicated list of exceptions and provisos).
--
Peter Rust
Jonathan, et. al,
> I've looked over the code, and committed some notes and some nit-picking corrections
Thank you – these are great, especially the missing “var” keywords. I also really appreciated the notes on how to find your way around the project – I was going to extend them a bit and add them to the wiki-generation script, but didn’t get to it this morning.
(Right now, the wiki-generation script is meshed in with the python example-tests in “overview_tests.py” because both rely on parsing the docstrings. In the near future, I’ll pull the docstring-parsing code out into utils.py, move the python testing code into “runtests.py” and move the wiki-generation code into a new file.)
> I didn't get the browser test to work. Instead I got (in Firefox) Access to restricted URI denied" code: "1012
This was probably Niall’s browser tests – the only tests I had written at the time were python tests in “overview_tests.py”.
This morning I wrote “runtests.py”, which uses a Javascript-multiline-comment-extraction function in utils.py to generate a set of Javascript tests (it places these in “tests/pylib_tests.json”). Then it fires up the default browser and points it at “tests/pylib_tests.html” which reads and executes the tests. I have some code in Javascript that supports multi-line doctests – I’ll bring this over to python. In the meantime, it just supports single-line tests (but I wrote 15 this morning and they provide good coverage of the feature I wrote this morning).
> my general philosophy is adapt the Python semantics that it fits in nicely with JavaScript,
> rather than aim for 100% compatibility (with a complicated list of exceptions and provisos)
I agree to a point. I don’t want the generated code to be an unreadable mess, so I’m ok with saying “py2js doesn’t support operator overloading”, as that would necessarily turn every operator into a method. There are other things that are just plain impossible – like lists evaluating to false if they’re empty. I’m ok with saying “you need to do ‘if len(my_list):” instead of just ‘if my_list:’ – in fact, I don’t see any way around it.
However, there are quite a number of well-implemented, well-known solutions for multiple-inheritance in Javascript. I think it would be valuable to take a well-written one (I would probably look most closely at Dean Edwards’ base2 implementation and Google’s Closure Library implementation) and use it.
On a similar note, I’ve been chewing on wrapping functions with def() to support kwargs and args. I implemented it this morning and was able to make all the following tests pass:
Wraps the supplied function in Pythonic goodness. The wrapper handles
unpacking **kwargs, unpacking *args, sending any extra arguments to the args
and kwargs parameters, sending this to the self parameter and setting the
__name__ property.
Use the $args() wrapper instead of * to send in positional args for unpacking.
If the combination of regular arguments passed and $args() passed are more
than the number of declared arguments, the extra ones will be passed as a
list to the "args" parameter. If there is no "args" parameter defined on the
function, an error will be thrown.
>>> var fn1 = def(function(arg1, arg2) { return [arg1, arg2].join(', '); });
>>> fn1('test1', 'test2');
"test1, test2"
>>> fn1($args(['test1', 'test2']));
"test1, test2"
>>> fn1('test1', $args(['test2']));
"test1, test2"
>>> fn1('test1', $args(['test2', 'test3']));
Error: __name__() takes at most 2 arguments (3 given).
Use the $kwargs() wrapper instead of ** to pass in keyword arguments for
unpacking. If any kwargs are passed in that don't map to a declared
parameters, the extra ones will be passed in a dict to the "kwargs"
parameter. If the function doesn't declare a "kwargs" parameter, an error
will be thrown.
>>> fn1($kwargs({'arg2': 'test2', 'arg1': 'test1'}));
"test1, test2"
>>> fn1('test1', $kwargs({'arg2': 'test2'}));
"test1, test2"
>>> fn1($kwargs({'arg3': 'test3'}));
Error: __name__() got an unexpected keyword argument 'arg3'
Here is an example of declaring "args" and "kwargs" parameters to catch
extra arguments:
>>> var fn2 = def(function(arg1, args, kwargs) { return arg1 + ', ' + JSON.stringify(args) + ', ' + JSON.stringify(kwargs); });
>>> fn2($args(['test1']));
"test1, [], {}"
>>> fn2($args(['test1', 'test2']));
"test1, [\"test2\"], {}"
>>> fn2($kwargs({'arg1': 'test1', 'arg2': 'test2'}));
"test1, [], {\"arg2\":\"test2\"}"
Normally in Javascript, I set defaults by simple one-liners at the top of a function: “if (my_var === undefined) my_var = 'default';”. Py2js could generate these (similar to your approach below), or could pass the defaults to the def() wrapping function.
I pushed my changes to bitbucket, you can run the browser tests by running “python runtests.py”.
-- peter
From: js...@googlegroups.com [mailto:js...@googlegroups.com] On
Behalf Of Jonathan Fine
Sent: Tuesday, April 27, 2010 2:07 PM
To: js...@googlegroups.com
Subject: Re: My first pass on py2js
Hello Peter
Jonathan Fine
Thank you for providing runtests.py. (I had to make a small change, now pushed, before it would work for me).
I don't have time tonight to discuss the other issues you raise.
Peter Rust
Jonathan,
I didn’t know about the webbrowser module – that’s great, thank you! (I had a suspicion the subprocess.Popen() trick might not work reliably).
-- peter
From: js...@googlegroups.com [mailto:js...@googlegroups.com] On
Behalf Of Jonathan Fine
Sent: Thursday, April 29, 2010 1:05 PM
To: js...@googlegroups.com
Subject: Re: My first pass on py2js
Hello Peter