PyInstaller: Finding imports of Cython extension modules

[David Vierra]

PyInstaller is a utility for freezing a Python script including all of
the modules recursively imported by that script along with their binary
dependencies. I'm helping to develop PyInstaller and I'd like it to have
better support for Cython.

PyInstaller finds imports using the modulegraph library, which scans the
ASTs or bytecode of pure-python modules and is able to locate and
include Cython extensions imported from pure python modules. However, it
is not able to scan a Cython extension for any Python modules (either
pure-python or compiled extensions) imported by that extension. I'm
searching for ways to make PyInstaller aware of these imports.

Is there a simple way to list the modules imported by a Cython extension?

My first idea is to find the original source files used to compile the
extension and scan those instead. For extensions compiled from .py
files, we can use modulegraph to scan the Python source code as usual.
If the extension was compiled from a .pyx file, I thought about using
Cython's own parser to build an AST from the Cython source and scan the
AST, looking for import statements.

To get the AST for Python source code, we use the built-in `compile()`
method. I looked at the Cython.Compiler.Parsing module briefly, but
couldn't find an equivalent function - it looked like I would have to
construct a Cython.Main.Context and then call parse() on it. I also
looked at the test suite and found the TreeFragment class which might be
able to return an AST for a source file.

What is the best way to call into Cython's parser and have it return an AST?



--
David Vierra
MCEdit, a Minecraft World Editor
http://www.mcedit.net
http://twitter.com/codewarrior0