pure Python parser for Python expressions

[Ondrej Certik]

Hi,

currently sympy has this nice feature that it can parse any Python
expressions and convert things like 1/2 into Integer(1)/Integer(2):

In [2]: sympify("1/2+x+y+sin(z)")
Out[2]: 1/2 + x + y + sin(z)

This is implemented in sympy/core/ast_parser.py using python ast and
it has only about 70 lines of code, so it's easy to maintain. However,
the modules parser, ast and compiler are not available on the google
app engine (even Jython has them though!):

http://code.google.com/p/sympy/issues/detail?id=871

so I reported it to the google folks here:

http://code.google.com/p/googleappengine/issues/detail?id=1468

but I doubt they are going to fix this.

Another problem is that the ast module can't handle large expressions:

http://code.google.com/p/sympy/issues/detail?id=976

(it raises "RuntimeError: maximum recursion depth exceeded").

so the only solution that I can see to fix all the problems above is
to write our own parser.

Google app engine supports eval(), so we may just write a simple
preparser based on the "re" module (also available on the app engine),
that converts things like 1/2 to Integer(1)/Integer(2) and "x" to
Symbol("x") and we should be fine. I don't know if eval() can handle
large expressions (to fix #976), that would have to be tried. Usually
a large expression is one big Add instance, so one solution that just
occured to me is to split it into several smaller Add instances,
evaluate each using eval() and add them together.

Ondrej

[Robert Kern]

On Sun, May 3, 2009 at 10:59, Ondrej Certik <ond...@certik.cz> wrote:

> Google app engine supports eval(), so we may just write a simple
> preparser based on the "re" module (also available on the app engine),
> that converts things like 1/2 to Integer(1)/Integer(2) and "x" to
> Symbol("x") and we should be fine. I don't know if eval() can handle
> large expressions (to fix #976), that would have to be tried. Usually
> a large expression is one big Add instance, so one solution that just
> occured to me is to split it into several smaller Add instances,
> evaluate each using eval() and add them together.

It's feasible to write and maintain a pyparsing parser for Python's
expression subgrammar. This also opens up the possibility for easily
extending the grammar to add other operators and such.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco

[Robert Kern]

On Sun, May 3, 2009 at 11:06, Robert Kern <rober...@gmail.com> wrote:
> On Sun, May 3, 2009 at 10:59, Ondrej Certik <ond...@certik.cz> wrote:
>
>> Google app engine supports eval(), so we may just write a simple
>> preparser based on the "re" module (also available on the app engine),
>> that converts things like 1/2 to Integer(1)/Integer(2) and "x" to
>> Symbol("x") and we should be fine. I don't know if eval() can handle
>> large expressions (to fix #976), that would have to be tried. Usually
>> a large expression is one big Add instance, so one solution that just
>> occured to me is to split it into several smaller Add instances,
>> evaluate each using eval() and add them together.
>
> It's feasible to write and maintain a pyparsing parser for Python's
> expression subgrammar. This also opens up the possibility for easily
> extending the grammar to add other operators and such.

Another alternative is Fredrik Lundh's hand-built parser here:

http://effbot.org/zone/simple-top-down-parsing.htm

It has the advantage of already being written.

[Ondrej Certik]

On Sun, May 3, 2009 at 9:30 AM, Robert Kern <rober...@gmail.com> wrote:
>
> On Sun, May 3, 2009 at 11:06, Robert Kern <rober...@gmail.com> wrote:
>> On Sun, May 3, 2009 at 10:59, Ondrej Certik <ond...@certik.cz> wrote:
>>
>>> Google app engine supports eval(), so we may just write a simple
>>> preparser based on the "re" module (also available on the app engine),
>>> that converts things like 1/2 to Integer(1)/Integer(2) and "x" to
>>> Symbol("x") and we should be fine. I don't know if eval() can handle
>>> large expressions (to fix #976), that would have to be tried. Usually
>>> a large expression is one big Add instance, so one solution that just
>>> occured to me is to split it into several smaller Add instances,
>>> evaluate each using eval() and add them together.
>>
>> It's feasible to write and maintain a pyparsing parser for Python's
>> expression subgrammar. This also opens up the possibility for easily
>> extending the grammar to add other operators and such.
>
> Another alternative is Fredrik Lundh's hand-built parser here:
>
>  http://effbot.org/zone/simple-top-down-parsing.htm
>
> It has the advantage of already being written.

Thanks for both suggestions. I played with both and it should not be
that difficult as I feared.

I still think that an easier option is to just write a preparse()
function, that takes a string, and returns a string:

>>> preparse("3/4 + x")
'Integer(3)/Integer(4) + Symbol("x")'

And then we just pass it to eval(). That should be easy to debug and
maintain. Just like the lambdify() works.

Then the question is, how to write the preparse() function, it seems
to me a simple regex based substitution should do the job, just like
in preparse_numeric_literals() in sage.misc.preparser.py.

Ondrej

[Fabian Pedregosa]

I had to do some preparsing for django-sympy, and I used regular
expressions, so might be worth to take a look at

http://git.sympy.org/?p=django-sympy.git;a=blob;f=dsympy/parser.py;h=7ce868f822a7ab3be73fd9214b103ccb84031d45;hb=HEAD



> Ondrej
>
> >
>