Mixed Python/Cython/C++ project, selective recompiling

[tanguy...@gmail.com]

Hi all,

I am writing a C++ simulator with a Python interface, I have therefore a project that has three parts:

• a C++ source compiled into cproject.so
  
• a Python module
• some Cython files to allow direct access to some of the C++ functions and objects from Python, compiled into cywrappers.so
  

I am compiling everything with Cmake, first compiling the whole C++ source into one .so file, then calling setup.py and compiling the cython wrappers.

However, for some reason, every time I modify the C++ source, I have to recompile the Cython wrappers also to get the changes.

My guess is that this means I'm doing something wrong and the whole C++ code is recompiled into cywrappers.so but I am not sure why.

Basically I want to be able to compile selectively the C++ source (the cproject.so) or the cywrappers.so depending on which source files I changed.

The setup file that I use defines my cython extension as follow:

mod_name = '@PY_NAME@'

extensions = Extension(
    mod_name + ".cywrappers",
    sources = ['@PYMODULE_DIR@/' + mod_name + '/cywrappers.pyx'],
    language = "c++",
    include_dirs = [
        '@PROJECT_SOURCE_DIR@',
        '@PYMODULE_DIR@/' + mod_name,
        np.get_include()
    ],
    extra_compile_args = [],
    extra_link_args = [],
    libraries = ['cproject'],
    library_dirs = ['@CMAKE_INSTALL_PREFIX@/@CMAKE_INSTALL_LIBDIR@'],
    runtime_library_dirs = ['@CMAKE_INSTALL_PREFIX@/@CMAKE_INSTALL_LIBDIR@']
)

where Cmake fills in the different variables.

So basically, is the C++ code recompiled in cywrappers.so and, if so, how can I avoid it and just link to cproject.so?
I hope this question was not answered already somewhere, but I could not find anything...

Thanks in advance,
Tanguy

[Robert Bradshaw]

As far as I know, distutils doesn't have a "re-link but don't
re-compile" option, but you could look at the output it produces when
running and run the linking command only manually.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[tanguy...@gmail.com]

So basically you mean that this is the correct way to do it and that it's normal that I have to recompile?

Could you tell me why the changes in the C++ are not taken into account even if cproject.so is recompiled?
Since I don't access all the C++ files in the cython project and I dont not specify them as sources in the Extension, I find it very hard to believe that the whole C++ code is correctly recompiled in cywrappers.so, my guess is thus that cproject.so is in some way incorporated to cywrappers.so during setup, am I correct?

Best

[Robert Bradshaw]

I'm saying this is how distutils does it. Generally cproject.so would
be dynamically linked into cywrappers.so. Note that if any of the
headers of your C++ project change, that may require re-compiling
cywrappers regardless (and in C++, often lots of implementation detail
goes into the headers).

[Tanguy Fardet]

Ok, thanks for the precision.

I finally managed to write something that seems to work: I check if the
cython files changed in Cmake and I do a setup.py install --skip-build
if they did not.

I suspect the problem also came from the fact that I am subclassing
build_ext, so the --skip-build argument was not working on its own at
first and I had to configure my subclass to be more "receptive".

Regarding my first message where I said that changes in the C++ files
were not taken into account if I did not recompile, I guess that wasn't
true in the end; probably what I was doing when "not recompiling" was
also skipping the linking step or something.
As you mentioned, I'm still not sure what is going to happen if the
cython files are untouched but one of the few C++ files that they import
from is changed, though... we'll see ^^

Best,
T