deployment issues

[bart]

I've developed a compiler using PLY, and I need to distribute it in an
RPM as part of a python package. I'd like some advice on a couple of
related issues.

First, I need to generate the parsetab and lextab .py and .pyc files
during the RPM build, and install them under /usr/lib/python/site-
packages/<package>.

Second, when the compiler is used, I need it to use these files (which
probably won't be in sys.path) instead of generating new ones in the
current directory.

I've seen in the documentation that you can change the name of the
parsetab and lextab files, and you can change the output directory of
the parser output files. So, I could probably cobble something
together. However, I'm hoping someone has already done this, or
otherwise has some advice on a clean solution.

Thoughts?

[bart]

Hmm. No replies yet. Maybe I'm the first to package up a PLY-based
compiler.

Here's what I've come up with so far. Suppose my compiler is part of
a package called "foo". In my compiler code I say:
yacc.yacc(tabmodule="foo.parsetab")
lex.lex(lextab="foo.lextab")

In my setup.py, I cause the parsetab and lextab files to be generated,
then I rename foo.parsetab.py to parsetab.py and foo.lextab.py to
lextab.py. I include parsetab.py and lextab.py in my MANIFEST file,
and install them with my RPM package as part of the foo python
package.

This is working, but is not as clean as I'd like. It relies on some
PLY implementation details, namely:
lex.py: exec "import %s as lextab" % tabfile
yacc.py: exec "import %s as parsetab" % module

Does anyone have a better solution?

[A.T.Hofkamp]

bart wrote:
> Hmm. No replies yet. Maybe I'm the first to package up a PLY-based
> compiler.

You may be the first that generates and saves the tables at build/install
time. I usually just suppress generation of the table files, and let the
program compute the tables on each run. Since it is normally a fraction of a
second, this solution is much easier imho.

However, I now have a grammar that needs 1.4 seconds for table generation, and
0.008 seconds for parsing a 'hello world'-like program.
The comparison is not entirely fair, since most programs will be larger, but
for this one I am afraid I'll have to do something like you did.

I am still collecting courage however....

> This is working, but is not as clean as I'd like. It relies on some
> PLY implementation details, namely:
> lex.py: exec "import %s as lextab" % tabfile
> yacc.py: exec "import %s as parsetab" % module

Technically, you could copy both files into your project, so you know for sure
this will not change without you knowing.
I don't know what the license says about this though.

If it is not allowed, you could check explicitly for the version of the PLY files.

Sincerely,
Albert

[Geert Jansen]

Hi,

both yacc.yacc() and lex.lex() have a "tabmodule" keyword argument that can be used to control the name of the generated tables. In my project Python-AD I use a script called "gentab.py" that creates the parser tables using this argument prior to distribution. As long as the future instantiations of yacc.yacc() and lex.lex() use the same module name, they will be able to find the parser tables.

I fully agree that parser tables should be generated prior to distribution and then be packaged with the application. Generating them on the fly wastes a lot of time.

If you want to look at my code, see:

 - http://www.boskant.nl/trac/python-ad/browser. Look at "gentab.py" and "lib/ad/util/parser.py"
 - my post name "OO API to lex/yacc" from december 2007.

Regards, Geert

- I have a script called "gentab.py" that generates the files