Avoid creation of parsetab in user's dir

[eliben]

Hello,

I'm packaging a library that uses PLY for distribution. The setup
script is placing the library directory "mylib/" into the user's site-
packages. Inside, there's a parser file that uses PLY
(mylib_parser.py), and I've also placed a fresh parsetab.py into it.

In user's code, when he imports:

from mylib import mylib_parser

And creates the parser, PLY starts generating the parser table and
places it into the user's current directory.

How can I hide this implementation detail from the user ? I don't even
want him to know that any parser is used internally.

Thanks
Eli

[Bart Whiteley]

This thread has some info about the problem you are trying to solve:
http://groups.google.com/group/ply-hack/browse_thread/thread/848eeedab7152f5b/8294806bafe22a57

If I recall correctly, the solution I ended up with (which is
different from the one described in the thread above) requires that
lex.py and yacc.py from the PLY distribution be bundled with my
module.

I also rigged my setup.py to generate the lextab and parsetab files,
so that they can be compiled and the corresponding .pyc files are
installed by setup.py.

You're free to examine my code:
http://pywbem.svn.sourceforge.net/viewvc/pywbem/pywbem/trunk/mof_compiler.py?view=markup
http://pywbem.svn.sourceforge.net/viewvc/pywbem/pywbem/trunk/setup.py?view=markup

[eliben]

On Oct 18, 9:22 pm, "Bart Whiteley" <bwhite...@gmail.com> wrote:
> This thread has some info about the problem you are trying to solve:http://groups.google.com/group/ply-hack/browse_thread/thread/848eeeda...

>
> If I recall correctly, the solution I ended up with (which is
> different from the one described in the thread above) requires that
> lex.py and yacc.py from the PLY distribution be bundled with my
> module.
>
> I also rigged my setup.py to generate the lextab and parsetab files,
> so that they can be compiled and the corresponding .pyc files are
> installed by setup.py.
>

> You're free to examine my code:http://pywbem.svn.sourceforge.net/viewvc/pywbem/pywbem/trunk/mof_comp...http://pywbem.svn.sourceforge.net/viewvc/pywbem/pywbem/trunk/setup.py...
>
>

Thanks. I would expect at least a quasi-standard way to exist, or
documented in some way in the manual. But if there's none, hacks like
this can be made to work, I see.

Eli

[eliben]

> This thread has some info about the problem you are trying to solve:http://groups.google.com/group/ply-hack/browse_thread/thread/848eeeda...

>
> If I recall correctly, the solution I ended up with (which is
> different from the one described in the thread above) requires that
> lex.py and yacc.py from the PLY distribution be bundled with my
> module.
>
> I also rigged my setup.py to generate the lextab and parsetab files,
> so that they can be compiled and the corresponding .pyc files are
> installed by setup.py.
>

> You're free to examine my code:http://pywbem.svn.sourceforge.net/viewvc/pywbem/pywbem/trunk/mof_comp...http://pywbem.svn.sourceforge.net/viewvc/pywbem/pywbem/trunk/setup.py...
>

I've looked at your code, and there's something I don't understand.
The _build function creates the tables and is called by setup.py, I
see.
But if a user imports your module, how does it find the tab files in
the path and doesn't regenerate them ?
Did you modify PLY for that ?
Eli

[Bart Whiteley]

No. As I recall now, it works simply because I pulled lex.py and
yacc.py from ply and included them in the same directory as my code.
When the code within lex.py and yacc.py tries to import the parsetab
and lextab modules, it first looks in the same directory where lex.py
and yacc.py reside. It finds the modules there. so it doesn't
regenerate them.

If you don't want to do that, you'll have to do something like what is
discussed in this email thread, mentioned previously:
http://groups.google.com/group/ply-hack/browse_thread/thread/848eeedab7152f5b/8294806bafe22a57?pli=1

Sorry for the confusion.

[eliben]

> > I've looked at your code, and there's something I don't understand.
> > The _build function creates the tables and is called by setup.py, I
> > see.
> > But if a user imports your module, how does it find the tab files in
> > the path and doesn't regenerate them ?
> > Did you modify PLY for that ?
>
> No.  As I recall now, it works simply because I pulled lex.py and
> yacc.py from ply and included them in the same directory as my code.
> When the code within lex.py and yacc.py tries to import the parsetab
> and lextab modules, it first looks in the same directory where lex.py
> and yacc.py reside.   It finds the modules there. so it doesn't
> regenerate them.
>
> If you don't want to do that, you'll have to do something like what is

> discussed in this email thread, mentioned previously:http://groups.google.com/group/ply-hack/browse_thread/thread/848eeeda...
>
> Sorry for the confusion.

I ended up using the following method:

My package is named pycparser, and in it the parser and lexer get
'pycparser.yacctab' and 'pycparser.lextab' into their parsetab and
lextab parameters. This makes sure that these tables are looked for in
the module's directory, where they're initially placed with the setup
script.

So *users* of the module need not see the table file, and yet they do
enjoy the optimizations.

People who want to modify the package, have the option to pass in to
the parser to disable yacc/lex optimizations, and they will have the
tables regenerated for each new run.

Eli

[D.Hendriks (Dennis)]

For a different discussion (also started by Eli):

  Topic: FAQ on optimized modes of lex and yacc
  Link: http://groups.google.com/group/ply-hack/browse_thread/thread/180e6fe2d3052c52/12adcdd411a0bff3?#12adcdd411a0bff3

I just posted a reply that I believe is also interesting for readers of this discussion:

    http://groups.google.com/group/ply-hack/msg/12adcdd411a0bff3

Dennis