Compiling the stdlib

[Stefan Behnel]

Hi,

for a lightning talk at PyCon-DE, and following a recent thread on
python-dev, I've written a setup.py script that compiles most of the
stdlib. For the latest developer sources of Py3.4, I managed to compile 612
out of 620 modules so far. Pretty cool if you ask me. The rest fails due to
compiler crashes. 8-]

You can get the setup.py script here:

http://consulting.behnel.de/cysetup.py

Note that you will need the latest Cython master because I added a build
mode to cythonize() that ignores modules that fail to build - makes sense
when compiling .py files optionally.

If you happen to have an application that makes heavy use of some stdlib
modules, please give it a try and report back if it works for you,
especially if you can come up with some benchmarks, that would be great.

Stefan

[francis]

Hi Stefan,
is it possible (or far away) to make it generate a static compiled
version of the stdlib together with a C-Extension? The idea is the I
would like to have line and branch coverage of a Python C-extension by
using gcc-coverage (gcov). AFAIN to be able to use gcov for that, the
extension module has to be compiled statically: means in that case the
interpreter, the tests (written in python with unittest) and the
C-extension itself. I wonder if that can be achieved with cython

Thanks in advance !

Francis

[Stefan Behnel]

francis, 31.10.2012 18:00:

> On 10/31/2012 05:39 PM, Stefan Behnel wrote:
>> I've written a setup.py script that compiles most of the
>> stdlib. For the latest developer sources of Py3.4, I managed to compile
>> 612 out of 620 modules so far. Pretty cool if you ask me. The rest fails
>> due to compiler crashes. 8-]
>

> is it possible (or far away) to make it generate a static compiled version
> of the stdlib together with a C-Extension? The idea is the I would like to
> have line and branch coverage of a Python C-extension by using gcc-coverage
> (gcov). AFAIN to be able to use gcov for that, the extension module has to
> be compiled statically: means in that case the interpreter, the tests
> (written in python with unittest) and the C-extension itself. I wonder if
> that can be achieved with cython

Yes, that's totally doable. In fact, we already do most of this in our
"cython_freeze" tool, although that rather targets executable main
programs. Worth taking a look.

Incidentally, I've recently discussed this with Martin von Löwis and we'd
both like to eventually see a couple of modules linked in as a build
option, above all the new importlib bootstrap module. That would provide an
advanced replacement for the original Python "freeze" tool.

Basically, all you need is some glue code that automatically registers the
statically linked extension modules with the Python runtime at startup.
CPython has the so-called "inittab" mechanism for that, which it also uses
for its current built-in modules. The actual list of modules, and in fact
the complete C glue code, could obviously be generated automatically from
the list of potentially required modules that the Python "freeze" tool
spits out.

Stefan

[Bradley Froehle]

Basically, all you need is some glue code that automatically registers the
statically linked extension modules with the Python runtime at startup.
CPython has the so-called "inittab" mechanism for that, which it also uses
for its current built-in modules. The actual list of modules, and in fact
the complete C glue code, could obviously be generated automatically from
the list of potentially required modules that the Python "freeze" tool
spits out. 

 

One small difficulty here is that inittab uses just the module name, not the complete package name, when looking up modules.  This places some (minor) restrictions on the name of your module; namely you cannot use any top level package name.

[Stefan Behnel]

Stefan Behnel, 31.10.2012 17:39:

Update: I've committed the script here:

https://github.com/cython/cython/blob/master/Tools/cystdlib.py

Usage example:

$ python cystdlib.py --current-python build_ext -i

However, I noticed that it's not trivial to figure out what will work and
what won't, even if a module compiles cleanly. One of the problems is
"inspect". In the current state of affairs, I don't see a way to trick it
into accepting Cython's own functions because it tests explicitly for an
object being a subtype of types.FunctionType, which is type(some_py_func).
That means that code using inspect will fail to handle Cython compiled
functions correctly.

Stefan