parser / translator in python

[lkcl]

i've spent the past few days translating the skulpt translator.js back
into python, and locating suitable standard python modules to replace
the ones in javascript in skulpt, and fixing a number of errors which
were missed (in translator.js). the code is here:
http://pyjamas.svn.sourceforge.net/viewvc/pyjamas/trunk/pgen/

regression tests can now be performed using test_parse.py, to verify
that the code works as expected, and i used this to locate and fix the
remaining bugs (by throwing the entire contents of the standard python
test suite source code at it) i located a pretty-printer and output
two dumps of the AST: one from the "standard" python suite "import
parser" and the other from pgen "from lib2to3.pgen import parse". all
but "bad_syntax2.py" and "bad_codec.py" successfully compile (and/or
produce the exact same error message).

"The Plan" is basically to use the standard pyjamas compiler to
bootstrap into pure python, in-browser. the reason why this wasn't
done with skulpt is, i suspect, that the compiler itself wasn't ready
to be used, in order to compile itself (optimally or otherwise), so
was written in pure javascript, instead. i much prefer being able to
perform tests in python (especially the direct comparison) so have
chosen to do the pure-python-then-bootstrap route.

in this way, pyjamas will get an in-browser compiler.

i've not integrated the code in, yet: it's stand-alone.

l.

[Scott Graham]

On Sun, Sep 13, 2009 at 11:55 AM, lkcl <luke.l...@googlemail.com> wrote:
>
> i've spent the past few days translating the skulpt translator.js back
> into python, and locating suitable standard python modules to replace
> the ones in javascript in skulpt, and fixing a number of errors which
> were missed (in translator.js).  the code is here:
> http://pyjamas.svn.sourceforge.net/viewvc/pyjamas/trunk/pgen/

Neat, sounds like this will be a good addition to pyjamas.

You mean transformer.js, right? I'm not clear on why you re-ported
though, does lib/compiler from Python2.6 not do what you need?

If you recall any of the errors you ran across in the .js version, I'd
be happy hear about them.

scott

[lkcl]

On Sep 13, 10:41 pm, Scott Graham <sgra...@gmail.com> wrote:

> On Sun, Sep 13, 2009 at 11:55 AM, lkcl <luke.leigh...@googlemail.com> wrote:
>
> > i've spent the past few days translating the skulpt translator.js back
> > into python, and locating suitable standard python modules to replace
> > the ones in javascript in skulpt, and fixing a number of errors which
> > were missed (in translator.js). the code is here:
> >http://pyjamas.svn.sourceforge.net/viewvc/pyjamas/trunk/pgen/
>
> Neat, sounds like this will be a good addition to pyjamas.

yes. it was on the [vague] roadmap i keep in my head, and it was the
amount of work involved in doing the parser that kept me from
starting. i'd initially looked at pypy's parser - i _just_ couldn't
separate the code from the rpython generator from the AST generator
from the this, the that - it didn't help that they had "TODO: must
separate the dependencies here" all over the place: in the end i gave
up.

so i'm reaallly glad you tackled it, in a sensible way, by using
lib2to3! damn good job.


> You mean transformer.js, right?

yes, and astgen, and ... well, i just patched bits together until it
worked :)

> I'm not clear on why you re-ported
> though, does lib/compiler from Python2.6 not do what you need?

well, i _hope_ it doesn't :) but - no, i don't believe it does,
because, for exactly the same reasons that you didn't use it, "import
parser" doesn't work, because it's written in c, not python. so -
lib/compiler depends on code that's written in c. so, what i did was:
port translator.js back into python, comparing it to python 2.5 lib/
translator.py, and kept the lib2to3 code (that you'd found and used)
pretty much as-is.

> If you recall any of the errors you ran across in the .js version, I'd
> be happy hear about them.

there were quite a few, and some omissions of functions (com_with and
a couple more). i just went hell for leather over the past few days,
on auto-pilot, knocking through them all until test_parse.py worked.

sorry!

a couple of approaches that you could consider taking:

1) get the skulpt compiler to the point where translator.py and
friends can be translated into javascript - with skulpt.

2) run through the same tests that i did (but under d8) and, when an
error occurs, take a peek at translator.py and back-port it (and the
missing functions).

one thing though: as i use python 2.5, not python 2.6, i did have to
back-track on some of the grammar rules, to python 2.5 grammar, in
order to get the exact same AST tree output from the [standard] python
"import parser" module.

surprisingly, though, translator.py hasn't changed much [from python
2.5 to 2.6] but i thought you'd best be aware of this.

l.