optimize=1 for yacc gets rid of p_error?

[blaxyzblaxyz]

When running the calc.py in examples/calc of the ply package, I enter
"3*(3)" (without the quotes) and get "9" as answer, which is correct.

When I enter "3*(*)" I get: "Syntax error at '*'", which is exactly
what the p_error function is supposed to do.

I then modify calc.py. I change "yacc.yacc()" to
"yacc.yacc(optimize=1)". I run the program again. I once again enter
"3*(*)". This time however, I get "yacc: Syntax error at line 1,
token=*" on my screen. It turns out that yacc no longer has an error
function defined and therefore falls back to printing something to
stderr itself. The reason for this is that since the parsetab.py was
loaded there is no longer a p_error present (internally, in yacc,
self.errorfunc = None).

In my application I encountered the same problem. I have a p_error
function that raises a custom parsing exception. However, since I use
optimize=1, yacc no longer has the error function and thus prints a
message to stderr and continues! This means that None is returned as
parse result and the rest of my code crashes, since it cannot handle
None as parse result. (Note that it should not have to handle None, as
parse errors should be handled by p_error and the code DOES handle the
custom parsing exceptions).

So, to conclude, using optimize=1 (using the imported parsetab.py),
yacc loses the p_error function and this changes the outcome of
parsing!

It may also be interesting to know that the same does not apply to
lex, for there the t_error function is preserved, even when using
lextab.py.

Finally, I wonder whether all of this is wanted behavior or not. In my
opinion it shouldn't be...

[blaxyzblaxyz]

The solution seems to be to move lines 2088 until 2106 (that is "#
Look for error handler" until "print >>sys.stderr, "yacc: Warning. no
p_error() function is defined."") to line 2197 (that is, just before
" # Made it here. Create a parser object and set up its internal
state."), while obviously unindenting them once.

I'm not sure if this breaks anything else, but it seems to work just
fine.

Is the someone who can tell me if the change is OK or if it breaks
anything? Also, could this fix be included in a future release? Thanks
in advance for any answer.

[David Beazley]

This is the first I've heard of this, but I'll definitely make sure it gets fixed in the 2.4 release.

Cheers,
Dave

On Mon 28/04/08 8:31 AM , blaxyzblaxyz blaxyz...@hotmail.com sent:

[blaxyzblaxyz]

That would be great. Thanks.

By the way, any thoughts on when version 2.4 will be ready to be
released?

[David Beazley]

Not really. I've been collecting a number of bugs over the last 4-5 months and fixing them. I was going to make some updates to the documentation, but I could
probably make a release at almost any time now (although I'm going to investigate this optimize bug first). The absolute latest version of PLY can always be found in the
subversion repository however (should be links on the main PLY page).

Cheers,
Dave

On Mon 28/04/08 9:57 AM , blaxyzblaxyz blaxyz...@hotmail.com sent:

[blaxyzblaxyz]

It seems the same problem (including an almost identical solution) was
already posted once: http://groups.google.com/group/ply-hack/browse_thread/thread/55a22c7dfc602411/3d6827f00229758b

[David Beazley]

I just looked at the PLY CHANGES file and don't see any mention of fixing this yet. I'll get on it as soon as I get a chance.

Cheers,
Dave

On Mon 28/04/08 10:06 AM , blaxyzblaxyz blaxyz...@hotmail.com sent:

[David Beazley]

I just checked in a patch for this into the SVN repository (should be linked off
of the PLY page).