Dependency compiling in cython

[SirVer]

Hi,

I try hard to get the following szenario working:

module/geometry/objects.pyx
module/geometry/objects.pxd
module/calc/converter.pyx <- uses objects.pxd
module/calc/converter.pxd
module/calc/user.pyx <- uses objects.pxd & converter.pxd
module/calc/user.pxd
module/reconstruct/reconstruct.pxy <- uses user.pxd

In an ideal world, changing objects.pxd should trigger a rebuild of
converter.
user and reconstruct. This is not an ideal world; i didn't even manage
to get this to compile automatically.

Things I tried
1) numpy.distutils, monkey patched to use Cython instead of pyrex [1].
This seemed to work great, but upon test running I get the following
problem while importing user in reconstruct.pyx:
ValueError: calc.converter.Converter does not appear to be the correct
type object. Now (manually) running:
& cython calc/convert.pyx -I .
& cython calc/user.pyx -I .
and rerunning python setup.py build_ext -i (which only compiles the c
files and doesn't run cython again) fixes this. The next time I make
some changes to any files, the problem comes back again.

2) using scons tools from the python-devel branch.
I have worked my fair share with scons before; also extending it with
tools; So i know my fair share about it. I added an SConstruct in
module and SConscripts in the submodules. I also hacked at the tool
because it did assign strange module names to the modules upon
compilation. I am willing to work on this solution, which currently
seems the most promising for me; nevertheless I realize that also this
solution is not able to track dependencies. I also had some
difficulties running my test suite:
ImportError: No module named converter
when trying to import converter in user when trying to import user in
reconstruct.

I desperatly need help. Aren't others working with highly dependant
cython modules? I would also invest some time to work on any build
system that is considered feasible.

[1] http://www.mail-archive.com/numpy-di...@scipy.org/msg20199.html

Thanks for reading and your help!
Holger

[Robert Bradshaw]

Unfortunately, as you found out, this doesn't work. The recursive
build option in Cython is broken, and which means the .dep files are
not generated or used. I'm not sure if the recursive option was ever
set up to work with distutils though, I think it was for running
Cython on the command line only.

> 2) using scons tools from the python-devel branch.
> I have worked my fair share with scons before; also extending it with
> tools; So i know my fair share about it. I added an SConstruct in
> module and SConscripts in the submodules. I also hacked at the tool
> because it did assign strange module names to the modules upon
> compilation. I am willing to work on this solution, which currently
> seems the most promising for me; nevertheless I realize that also this
> solution is not able to track dependencies. I also had some
> difficulties running my test suite:
> ImportError: No module named converter
> when trying to import converter in user when trying to import user in
> reconstruct.

I'm not sure about this option.

> I desperatly need help. Aren't others working with highly dependant
> cython modules? I would also invest some time to work on any build
> system that is considered feasible.

In Sage, which has hundreds of dependant cython modules, we use our
own dependancy tracking system (see line 460+ of http://hg.sagemath.org/sage-main/file/tip/setup.py
) It also prioritizes what to build first, allows for parallel
cythonization, and runs an O(n) rather than the obvious O(n^2)
algorithm to get the set of modules that need to be rebuilt. This is
probably overkill for what you need... Also, it uses regex rather than
parsing to discover the dependancies, which is fast but not as clean.

I agree the situation should be improved. I'm not sure I'm sold on the
idea of requiring and cluttering the directory with .dep files. I'm
all ears for proposals. Perhaps the build_ext command could run
through the list of extensions, mark the ones that need rebuilding,
and then the .pyx -> .c phase would look for this marker.

- Robert

[SirVer]

Hi Robert, all,

A short introduction to this post:
I have a lot of experience with build systems. I worked extensively
with autotools, waf and scons for c/c++ projects and with scons/waf
for latex. I worked a bit with jam and cmake for cross platform stuff.
So far None of them got it *right*. A few days ago I was again
dissatisfied with scons for a simple task (building a PDF, copying it
from the build dir to the current dir, updating the file in a viewer
by sending a message to it) and I also didn't get my cython extensions
to build correctly (therefore this thread). I also toyed for a while
to try building a build system by memoization instead of dependency
trees. I was at the point to roll my own solution; then I found mem[1]
which was a build system by memoization started by someone else
programmed in python. I took it, bended it to my needs, added a few
unit tests here and there and have now a working solution for my
needs.

Which brings me back on topic: If you want to correctly build your
cython extensions, including dependencies tracking (of other pyx/pxd
or .h files) you can give it a shot. The corresponding makefile
(called MemfileRoot) looks like that:

----
# MemfileRoot
from mem.tasks.python import python_ext

def build():
python_ext("objects", "objects.pyx", inplace=True)
----

You can get a working copy from my working tree at github. It contains
an example for building python extensions via cython among others.
Setup should be easy:

$ git clone git://github.com/SirVer/mem.git
$ cd mem; sudo python setup.py install
$ cd examples/cython
$ mem
$ python -c 'import hello'
try changing any of the files and see it it rebuilds.

I am happy to provide feedback. Note that mem is rough around the
edges and not a full featured build system; but it does the job right.

I would be happy to hear feedback.

Cheers,
Holger

Ps:
I extracted the dependency tracking from sages setup.py; I also
found a few bugs in it( for example it doesn't track cimport hello as
h correctly).

[1] http://srp.github.com/mem/

On 11 Nov., 07:22, Robert Bradshaw <rober...@math.washington.edu>
wrote:

> own dependancy tracking system (see line 460+ ofhttp://hg.sagemath.org/sage-main/file/tip/setup.py