Upcoming features in Pyjaco 2.0

[Christian Iversen]

Hello again, world :)

I thought it'd take the time to write a separate mail about what some of
the new and planned features for Pyjaco 2.0 are. Of course, Pyjaco 2.0
isn't quite release-ready yet, but I'll let you know when we are closer.

### Finished ###

# IST core

The IST compiler design was agreed upon at the last dev meeting, and
it's finally here. The basic idea in the IST branch, is that you
construct a compilation pipeline with different elements. This is much
like a unix pipeline, where small programs perform simple tasks
cooperatively. We have 3 classes: Parsers, Transformers and Printers.
They responsible for input, transformation, and output, respectively.

Currently, we have the following elements:

- istcompiler.py
Really should be called PyCompiler, this uses the ast module to convert
python sourcecode into IST nodes.

- pyprinter.py
This module outputs an IST tree into Python. It's good for checking that
the IST correctly describes the input, since parsing and then printer
the python source code should be a no-op, unless there are errors.

- jsprinter.py
This module prints an IST tree in JavaScript syntax. If you use any
feature that *syntactically* does not make sense in javascript, it
complains. It does *not* worry at all about semantics. IST is a
Syntax-Tree after all :-)

- jsfier.py
This is the pyjaco core, where the magic takes place. It's a
re-implementation of the old python.py from pyjaco 1. But much, much
better :-)

# New method call arg passing

Method calls are much cleaner now. Whereas they required up to 4
function calls and 1-2 object constructors before, they now only require
1 function call and 0-2 object constructors. It depends on the amount of
advanced features you require (*varargs and **kwargs are more expensive)

# New attribute lookup semantics

I've been working on making attribute lookup semantics much closer to
pythonic. I've removed the default implementations of object.__getattr__
and object.__setattr__. The compiler now defers this to $PY.getattr(obj,
key) and $PY.setattr(obj, key, value). This mirrors how CPython and PyPy
do this internally.

# Support for multi-term

Due to lack of celebrity (and, honestly, lack of oversight) the pythonic
multi-term comparisons were not supported earlier. This includes (a < b
< c), but in the mathematical sense, where it means (a < b) && (b < c),
but where the values are only computed as needed, and never more than once.

Multi-terms now work in Pyjaco 2.0, including terms with side effects.

# Support for orelse-blocks

In Pyjaco 1.0, you couldn't reliably do try-else, while-else or
for-else. These statement types now all work, and have gained
corresponding unit tests.

# Proper support for type comparison

In Pyjaco 1.0, you could actually cause a crash-to-js by doing __eq__ on
certain built-in types. It wasn't pretty, but it's now fixed and has a
corresponding unit test.

# The stupid bug: list.pop()

It turns out, pyjaco's list.pop() never looked at the optional argument.
So list.pop(0) did the same as list.pop(). Oops. There's a unit test for
that now!

# List comprehension

List comprehension is now fully supported, and can use any nesting level
and number of conditions.

# Dictionaries from iterables

Due to some implementation details in the old pyjaco stdlib, it was
impossible to make dict instances from custom iterable types (classes
with __iter__). This is now fixed, and has a unit test.

# Syntax output quality

It's no secret that pyjaco now outputs faster, better javascript than
before, but it also does so while maintaining correct indentation, and
the correct number of semicolons! This makes the code easier to read and
debug, which is a Good Thing (tm)

### In progress ###

# Descriptor protocol

Descriptor protocol is one of the most internal, least well-known
features of python (for more info:
http://docs.python.org/2/howto/descriptor.html)

They are also incredible powerful, and necessary to understand to
emulate them :-)

I'm currently contemplating various ways we implement this, that wont
mean a massive speed penalty to every other part of the code. I have a
feeling there is a reasonable way, but I don't have all the details yet.

# True types

In python, you could argue there are 3 modes of object: metaclass, class
and instance. When

class Foo(object):
x = 42

That is really the same as saying

Foo = type("Foo", (object,), dict(x = 42))

Where type() is used in the 3-argument form, where it creates new
classes. This is fairly advanced stuff, but if we can implement true
types and the right attribute semantics, they would be easy to implement
in Pyjaco. At that point, metaclasses would also be easy.

# Correct attribute semantics

This is in connection with true types. It is very, very tricky to map
certain python features, particularly the combination of
__getattribute__, __getattr__, __dict__, parent lookups and instance
lookups into one reasonably fast implementation. I'm still working on
it, though.

# Multiple inheritance

If we had correct attribute semantics, multiple inheritance would be
just around the corner. And in fact, it would make quite a few things
simpler and more pythonic, since less hacks would be required.

# Module support

I know, mainline has Gordons module support, but the rewritten version
doesn't have that yet. Gorden, maybe you have time to help integrate it?
I think many of the module improvements to mainline are quite closely
applicable in the new branch.

### Ideas ###

# Optimization transformers
Since a transformer type pipeline piece can inspect and change only the
parts it cares about, it would be interesting to have a small-ish
transform module with optimizations. I have a quite a few ideas for
recognizing certain idioms and rewriting them to use a faster way to do
the same thing.

For example, when constructing a dictionary object (of immutables), a
lot of the javascript evaluation could be pre-computed, and stored
directly in the source. Particularly, the hash() calculation for each
element.

# Debug transformer
We could make a transformer that inserts a Comment() node for each
original line of source code, to help debugging. Or one that updates
filename/line number state, so that backtraces actually provide useful
information.

--
Med venlig hilsen
Christian Iversen

[Samuel Ytterbrink]

Aweeeeeesome!

--
--
You are subscribed to the Google Group pyj...@googlegroups.com
To unsubscribe from this group, send email to
pyjaco+un...@googlegroups.com

--- You received this message because you are subscribed to the Google Groups "pyjaco - The Python to Javascript compiler" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyjaco+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
//Samuel Ytterbrink

[g]

# Module support

I know, mainline has Gordons module support, but the rewritten version

doesn't have that yet. Gordon, maybe you have time to help integrate it?

I think many of the module improvements to mainline are quite closely
applicable in the new branch.

 

I am really busy right now, but I will make some time in the near future to learn how the internals of the new IST based pyjaco work and connect together.  I am on board with getting module support into Pyjaco 2.0.  Great work so far Christian

[Christian Iversen]

Excellent! Just let me know any time if you want to learn more.

I can tell you already that the design was a huge success. I know we had
planned a few more pipeline stages, and while it certainly be possible
to split the (current) middle pipeline part into more parts, I'm not
convinced it would help maintainability or readability.

This is the way to generate pyjaco code now:

pycode = istcompiler.Compiler().compile(
file("test.py").read(), "test.py")
jscode = jsfier.Transformer().compute(code)
pyjaco_output = jsprinter.format(code)
print pyjaco_output

That is _all_ that is required.

> Great work so far Christian

Thanks :)