Support for combining multiple PYX files in a single module
Ray Speth
Hi,
I have some questions about combining multiple separately-compiled Cython Pyx files into a single extension module. We have been doing this for a while now in Cantera to enable parallel builds and incremental rebuilds of the Cython wrapper for our core C++ library. We have ~15,000 lines of Cython code, and the builds were quite slow using the older approach of just creating a parent pyx file with "includes" for all the sub-modules.
I got this process working mainly thanks to this thread on StackOverflow. However, it has been noted that this approach is now leading to “one definition rule violations” in a couple places in our usage, so I have a few questions:
- Would it be of interest to support this approach in a more official capacity? I think this could be at either the level of fixing these ODR issues, adding some tests, and documenting what works now, or it could extend to providing a simpler interface for compiling multiple Pyx files into a single module. I’d be happy to contribute with a bit of guidance.
- Is there another approach I should consider? I think refactoring our our interface as 18 separate extension modules, all linked to the shared C++ library and with various links between the modules, would be fairly tricky.
Regards,
Ray
da-woods
Hi Ray,
As far as possible, everything internal with Cython (i.e. things we define starting with `__pyx_`) is defined as `static` so it shouldn't suffer from one-definition issues. However that doesn't cover struct definitions so they could cause problems (as __pyx_mstate seems to). In C++ we could put them in an anonymous namespace and that'd solve your problem. If you wanted to open an issue (or even better, a PR) to do that then that's fine by me. I think C++ is probably less forgiving than C here.
Some of your other problems look like they come from Fortran rather than Cython (so I'm not sure we could realistically fix it).
If you did want to contribute then I think what you need to do is:
* identify some/all of problematic structs.
* work out what code generates them ("git grep" might help;
failing that then put a conditional breakpoint in
CCodeWriter.write to catch it)
* wrap them in `#ifdef __cplusplus\n namespace {\n #endif`
* create a srctree test that demonstrates it - copy an existing
one, but essentially you bundle multiple files into one to test
linking multiple things together
(I'm happy to help in more detail if you get part way through this and get stuck).
I've personally been slightly reluctant to get involved in this because it feels like a slippery slope to trying to support "create a bundled compiled executable with all your Python dependencies" which feels like a direction where madness lies. But equally we don't want to impede you if you find this build approach works for you so some level of support and testing seems appropriate - you aren't the only ones doing this.
David
--
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cython-users/dfb489ea-faa4-45c9-9c45-127c71b6fd9an%40googlegroups.com.