Native parser ready

[Jesse Millikan]

TL;DR: I believe the native parser (at https://github.com/jmillikan/lambda-py/tree/lex-ragg) is ready to start using, if someone wants to merge it. It requires Danny Yoo's parser framework Ragg (https://github.com/dyoo/ragg). I just pulled brownplt/master, so lex-ragg should merge back in quietly, and everything *should* work as before, including all invocations of python-main. I will be around to yell at since it's likely I've screwed up *something*.

As tested, it works 'the same' - output matches or both parsers fail - as the python 3.2 parser for all python files in tests and base/pylib, except one. On tests/scope/nonlocal-from-class-body.py, the python parser checks for existence of nonlocal variables. The native parser does not, which I assume is what's wanted. Features not yet implemented *and* not used in the tests include argument annotations and multiple slice indexes. The python parser is still available in case the native parser proves broken or too slow for some case (e.g. modules/unittest/test_builtin.py when string representation is resolved).

New options in python-main (as far as I know, all current invocations will still work 'as-is'):

--native-parser: use native parser (default)

--python-parser: use python parser (requires --python-path)

--test-parser: Perform a noisy comparison of the two parsers on stdin (requires --python-path)

--test-native-parser: Perform a quieter comparison (no output -> all match) on a test directory (require --python-path)

To support both parsers, occurrences of parse-python/port have been replaced with the parser parameter from "parser/parser.rkt": ((parser) port) instead of (parse-python/port port python-path). The native parser calls the lexer directly.

On ragg and performance: The ragg master works fine on the current tests, but I'm using a slightly mangled version with better performance on (at least) long tuple/array literals. I've provided a .plt for this in case it is needed - let me know if this is not packaged right. (see bug report on ragg for details.)

[Joe Gibbs Politz]

Cool stuff! I'm going to have a look and merge things in tonight or tomorrow.

> --
> You received this message because you are subscribed to the Google Groups
> "lambda-py" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lambda-py+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Joe Gibbs Politz]

Actually, can someone who doesn't already have `ragg` installed give
this a whirl? I'm using ragg for another project, so I'm not sure if
I tested the installation correctly. You should just be able to run

git pull git://github.com/jmillikan/lambda-py.git lex-ragg

From a clean checkout of lambda-py, and see what's missing when you try to

raco make python-main.rkt

I'm trying to decide if it's best to include ragg as a git submodule
of lambda-py, to require it from planet, or to just ask users to
install a .plt file.

[Matthew Milano]

We're already asking users to install a .plt file for the language
itself, so it strikes me that an additional .plt file would add no
additional end-user overhead.

[Joe Gibbs Politz]

OK, everything passed on my machine, so I'm going to push.

If you install ragg (plt file in the base directory), you should now
be able to run --test/--interp without using --python-path. I updated
the README to have some instructions for doing that from the command
line; you can also use DrRacket's interface.

[Jesse Millikan]

Joe, how would you prefer I contribute changes in the future, pull requests? (Assuming they don't require a lot of discussion, re-tooling, etc.)

[Joe Gibbs Politz]

I can add you to committers on the main repo if you want. If you're
making non-breaking changes, I'm happy to just have things added
incrementally. For example, I'm happy to do this with changes to the
parser going forward; we have ~6 committers so conflicts seldom come
up.

Big separate features like this parser are good to do via pull
requests on GH though, if you're working on more
experimental/separable things.

On Sat, Apr 13, 2013 at 8:03 PM, Jesse Millikan

[Joe Gibbs Politz]

Nice job testing and implementing to an interface by the by; this was
a pretty sweet and painless drop-in replacement!

[Alejandro Martinez]

Excellent, congratulations!

2013/4/13 Joe Gibbs Politz <joe.p...@gmail.com>

--
Alejandro.

[Alejandro Martinez]

I'm getting an error parsing this generator using --native-parser, which is parsed correctly using --python-parser:

def g():

  for x in range(5):

    if x % 2 != 0:

       yield x

print(list(g()))

this is the error:

adjust-indent-stack: arity mismatch;

 the expected number of arguments does not match the given number

  expected: 3

  given: 2

  arguments...:

   '(0)

   0

  context...:

   C:\Users\Ale\Documents\GitHub\lambda-py\base\parser\python-lexer.rkt:362:11:

slurg

Alejandro

[Jesse Millikan]

Whoops, stupid mistake. I've pushed a fix to master. I should build some better tests around the lexer edge cases.

[Alejandro Martinez]

Thanks Jesse, it's working now.

2013/4/19 Jesse Millikan <jesse.a....@gmail.com>

--

You received this message because you are subscribed to the Google Groups "lambda-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lambda-py+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Alejandro.