How to share C++ code between two Cython modules in the same package

[gmc444]

Hello,

I'm trying to build a Cython-based extension that contains two modules, both of which are a combination of Cython and C++ code. I'm having trouble getting it to build, and I'm hoping someone has suggestions for correcting this.

Module A1 contains a1.pyx and a2.cpp, module B2 contains b2.pyx and b2.cpp, with appropriate pxd and .h files. The code in a1.cpp has references to the code in b2.cpp, and the code in a1.pyx references b2.pyx (B2 is a library built by someone else; I have full access to the source but limited flexibility in how I can change it.)

When I treat A1 and B2 as separate extensions, then the shared lib build for A1 doesn't get built with a reference to the B2.so file, and I get unsatisfied externals when Python tries to import A1. I can manually work around this by patching the link directive for A1.so to reference B2.so, but I don't see how I can reliably do this from the setup.py script.

Alternatively I've tried treating A1 as a single extension and pulling in b2.pyx as an additional source file, but doing this gives me a warning about multiple .pyx files for the package, and when I attempt to import A2, I get an error where it's not able to resolve Python references into B2.

I'm using Python3.6,  on Linux, with Cython 0.26 and setuptools.

Thanks,

Greg

[Hai Nguyen]

On Wed, Aug 9, 2017 at 6:55 PM, gmc444 <gmc...@gmail.com> wrote:

Hello,

I'm trying to build a Cython-based extension that contains two modules, both of which are a combination of Cython and C++ code. I'm having trouble getting it to build, and I'm hoping someone has suggestions for correcting this.

 

Module A1 contains a1.pyx and a2.cpp, module B2 contains b2.pyx and b2.cpp, with appropriate pxd and .h files.

 I don't have answer for your issue but you should rename "b2.pyx" because it will be generated to "b2.cpp", which overwrite

your existing "b2.cpp"

Hai

The code in a1.cpp has references to the code in b2.cpp, and the code in a1.pyx references b2.pyx (B2 is a library built by someone else; I have full access to the source but limited flexibility in how I can change it.)

When I treat A1 and B2 as separate extensions, then the shared lib build for A1 doesn't get built with a reference to the B2.so file, and I get unsatisfied externals when Python tries to import A1. I can manually work around this by patching the link directive for A1.so to reference B2.so, but I don't see how I can reliably do this from the setup.py script.

Alternatively I've tried treating A1 as a single extension and pulling in b2.pyx as an additional source file, but doing this gives me a warning about multiple .pyx files for the package, and when I attempt to import A2, I get an error where it's not able to resolve Python references into B2.

I'm using Python3.6,  on Linux, with Cython 0.26 and setuptools.

Thanks,

Greg

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Barker]

hmm, I"m a bit surprised, I thought Linux would let you do this.

though you do need to make sure they get imported in the right order -- i.e. have it all in a python package that imports b before a

this has worked for us on OS-X and Linux

But it didn't work on Windows.

So the more robust solution is to compile all the (non-generated) C++ code into a shared lib, and then have each of the Cython extensions link to that.

On the other hand, if you only have two files, why not put A and B into a single cython module -- it would make this all easier.

It's also possible to have a single extension so that is a package with multiple modules in it -- though I did that years ago with hand-written code, I don't think anyone has adapted Cyton to support that.

-CHB

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[gmc444]

To clarify for Hai, the modules separate the pyx and cpp files into different directories, so overwriting isn't a concern.

I like the idea of putting the C++ code into it's own .so. I'll look into going that direction.

Thanks!

Greg

On Thursday, August 10, 2017 at 12:29:32 PM UTC-7, Chris Barker wrote:

hmm, I"m a bit surprised, I thought Linux would let you do this.

though you do need to make sure they get imported in the right order -- i.e. have it all in a python package that imports b before a

this has worked for us on OS-X and Linux

But it didn't work on Windows.

So the more robust solution is to compile all the (non-generated) C++ code into a shared lib, and then have each of the Cython extensions link to that.

On the other hand, if you only have two files, why not put A and B into a single cython module -- it would make this all easier.

It's also possible to have a single extension so that is a package with multiple modules in it -- though I did that years ago with hand-written code, I don't think anyone has adapted Cyton to support that.

-CHB

On Wed, Aug 9, 2017 at 3:55 PM, gmc444 <gmc...@gmail.com> wrote:

Hello,

I'm trying to build a Cython-based extension that contains two modules, both of which are a combination of Cython and C++ code. I'm having trouble getting it to build, and I'm hoping someone has suggestions for correcting this.

Module A1 contains a1.pyx and a2.cpp, module B2 contains b2.pyx and b2.cpp, with appropriate pxd and .h files. The code in a1.cpp has references to the code in b2.cpp, and the code in a1.pyx references b2.pyx (B2 is a library built by someone else; I have full access to the source but limited flexibility in how I can change it.)

When I treat A1 and B2 as separate extensions, then the shared lib build for A1 doesn't get built with a reference to the B2.so file, and I get unsatisfied externals when Python tries to import A1. I can manually work around this by patching the link directive for A1.so to reference B2.so, but I don't see how I can reliably do this from the setup.py script.

Alternatively I've tried treating A1 as a single extension and pulling in b2.pyx as an additional source file, but doing this gives me a warning about multiple .pyx files for the package, and when I attempt to import A2, I get an error where it's not able to resolve Python references into B2.

I'm using Python3.6,  on Linux, with Cython 0.26 and setuptools.

Thanks,

Greg

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.