Would anyone miss table file output in PLY?

120 views
Skip to first unread message

David Beazley

unread,
Apr 26, 2012, 2:27:21 PM4/26/12
to ply-...@googlegroups.com, David Beazley
I've been thinking about some cleanup to PLY lately and have a question. Would anyone actually miss the table output in PLY (parsetab.py, lextab.py) if it were removed entirely? Here's some context:

The original version of PLY was developed on a 200 MHZ home computer about 12 years ago. At that time, generation of the LALR tables was relatively expensive so creating table files was a way of caching the result and reusing it (much like Python uses .pyc files). Today, it's a bit different. For instance, making the LALR tables for the ANSI C grammar (over 200 rules and nearly 350 states) on a modern machine only takes around a half a second. Thus, I'm honestly wondering if I could just ditch all of that table reading/writing code and not worry about it.

Does anyone have any particular thoughts about this?

Cheers,
Dave

Erez Sh

unread,
Apr 27, 2012, 12:48:57 AM4/27/12
to ply-...@googlegroups.com
Half a second is quite a while, imho. ply can be useful for many types
of software, not just compilers, and a delay in the loading time can
really affect their usability.
As an alternative, perhaps you could make the table object pickle-able
(and accessible), so that the caller can implement caching himself if
he finds it important. As a bonus, it could be stored elsewhere, like
in a database. Just an idea.

Bart Whiteley

unread,
Apr 27, 2012, 1:01:19 AM4/27/12
to ply-...@googlegroups.com
I agree with Erez. I would like to see this optimization remain in some form.
> --
> You received this message because you are subscribed to the Google Groups "ply-hack" group.
> To post to this group, send email to ply-...@googlegroups.com.
> To unsubscribe from this group, send email to ply-hack+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ply-hack?hl=en.
>

David Beazley

unread,
Apr 27, 2012, 7:46:49 AM4/27/12
to David Cournapeau, David Beazley, ply-...@googlegroups.com
I agree that focusing on pickling might be a sensible way to go about it. Specifically, make it so that the instance returned by lex() and yacc() can be pickled as desired by the user.

I'm not planning any changes along these lines for Ply-3.5, but will consider it in future releases later.

Cheers,
Dave


On Apr 27, 2012, at 4:49 AM, David Cournapeau wrote:
> I use ply to parse the package declaration in my packaging tool bento, and the related command line tools would be significantly slower without the ply caching.
>
> OTOH, the current caching solution is a bit difficult to use in those situation: I think having a pickable object, and extracting the caching policy away from the main code would cover most usecases in that context.
>
> David

Gašper Ažman

unread,
Apr 26, 2012, 9:32:53 PM4/26/12
to ply-...@googlegroups.com
Hm.

I'm not using that functionality personally - as far as I'm concerned,
you can ditch it.

Gasper
> --
> You received this message because you are subscribed to the Google Groups "ply-hack" group.
> To post to this group, send email to ply-...@googlegroups.com.
> To unsubscribe from this group, send email to ply-hack+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ply-hack?hl=en.
>



--


Gašper Ažman

Rob Fowler

unread,
Apr 27, 2012, 8:02:13 AM4/27/12
to ply-...@googlegroups.com

Pickling sounds like a great idea, if possible. Personally the table system works ok for my compiler as well.

David Cournapeau

unread,
Apr 27, 2012, 5:49:37 AM4/27/12
to ply-...@googlegroups.com, David Beazley
On Thu, Apr 26, 2012 at 7:27 PM, David Beazley <da...@dabeaz.com> wrote:

Dave Benjamin

unread,
Apr 27, 2012, 2:01:23 AM4/27/12
to ply-...@googlegroups.com
I think making it so that the user can decide how best to store the precalculated data, or not to store it all, would be nicest. Dropping files in the current directory is not really ideal as a factory setting, though there are surely still use cases that benefit from that behavior.

Joseph S. Tate

unread,
Apr 27, 2012, 11:19:46 AM4/27/12
to ply-...@googlegroups.com
Another vote for leaving the optimization in some form.
> --
> You received this message because you are subscribed to the Google Groups "ply-hack" group.
> To post to this group, send email to ply-...@googlegroups.com.
> To unsubscribe from this group, send email to ply-hack+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ply-hack?hl=en.
>



--
Joseph Tate
Personal e-mail: jtate AT dragonstrider DOT com
Web: http://www.dragonstrider.com

John Szakmeister

unread,
Apr 26, 2012, 2:50:01 PM4/26/12
to ply-...@googlegroups.com
One project I worked on was almost as big as the ANSI C grammar. I
don't think it would have been quite as useful had we needed to wait
for tables every time. OTOH, processors have come a long way since
then, and I haven't worked on that project in a while. Personally, I
think I'd want to keep it from doing that work over again. But I
certainly wouldn't cry about it leaving either. :-)

-John

Boriel

unread,
Apr 27, 2012, 3:49:19 AM4/27/12
to ply-...@googlegroups.com
I agree with Erez and Bart.

BTW, I currently notice some delay when starting ply (my compiler, http://zxbasicand) for the first time it generates LALR tables, and they might be useful for other purposes.

On the other hand, I understand you want to clean up PLY (e.g. this makes it simpler and more maintainable). Maybe you could move that code out, to a different module or something?

Regards,
Jose

2012/4/27 Bart Whiteley <bwhi...@gmail.com>



--
Boriel      http://www.boriel.com

Reply all
Reply to author
Forward
0 new messages