make shared library for Cython and Python/C extensions?

[Andrew Dalke]

Hi all,

  Short version: how do I configure things so I can build a shared library that my traditional Python/C extension can use, as well as any new Cython extension(s)?

Long version, in case my guess of what I need is wrong.

I have a Python package which includes a C extension for performance reasons. It has 6kLOC of Python/C API code, and 25kLOC of C code. Currently my setup.py Extension() builds a single shared Python-importable extension (call it "_chemfp.so").

I want to migrate it to Cython. It's too tedious to re-write all of that code from a C extension to a Cython extension, so I want to do it in stages, over multiple releases. I also want to break that 6kLOC file into smaller submodules (eg, "_chemfp_bitops.so").

The 25kLOC of C code can be built as a shared library ("libchemfp.so"), with no Python in it. I would like to configure setup.py to build that shared library, have the Python/C extension "_chemfp.so" use that shared library, and have any Cython extensions, like "_chemfp_bitops.so", also use the same shared library.

And I want the result to still be pip-installable, as the current one is now; not have one mechanism to install the libchemfp.so shared library and another to install the Python extensions.

Thing is, I can't figure out how to do what I would like.

The only solution I've come up with is to have no sharing - build a copy of libchemfp.so into each package. This would multiply my build time from 1 minute (on the most optimized setting; 20 seconds on the least). The build time is already affecting my ability to develop the package, and one of the reasons for wanting to built several submodules, so I don't want to go that route.

Any suggestions or pointers?

        Andrew

        da...@dalkescientific.com

[Andrew Dalke]

On Thu, Aug 12, 2021 at 10:46 PM Andrew Dalke <andre...@gmail.com> wrote:

 Short version: how do I configure things so I can build a shared library that my traditional Python/C extension can use, as well as any new Cython extension(s)?

I've figured out one solution. See https://hg.sr.ht/~dalke/abcd for my final worked-out example.

In short, I put the C shared library code into my "core" module, along with the PyInit code for an empty module so I could leverage Python's importer.

I then developed a Python/C extension and a Cython extension, both using functions from that core shared library. So long as the shared library is imported first, everything works!

I was pleasantly surprised how simple the setup.py was. I didn't have to do anything special.

Regards,

        Andrew

        da...@dalkescientific.com