Sharing cython functions between modules: how to make it work (with a bit of project structure)

[David Poensgen]

Hello cython community,

I am struggling with sharing a set of cython (cdef-) functions between cython modules.

Unfortunately, the ressources I can find for this are rather sparse/only cover the most basic examples.

Briefly, the project structure, because I assume it might matter:

root folder / has setup.py

then 

/folder0/folder1 

contains a bunch of cython extensions which are compiled (in that same folder1 / module) when running setup.py 

extension0.pyx

extension1.pyx

_shared_ct.pyx

Specifically, I the latter file, _shared_ct.pyx contains a bunch of functions I want the other modules to have access to.

An example of the typical signatures:

 

cdef np.ndarray[np.float64_t, ndim=3] example_func(np.ndarray[np.float64_t, ndim=1] a, double[:,:,::1] b, int c, int d, int [:,::1] e, int f, bint g):

    cdef np.ndarray[np.float64_t, ndim=3] some_array
    # ... do lots of good stuff ...
    return some_array

I understand I need a _shared_ct.pxd file that contains the function signatures for that. So I have a file, _shared_ct.pxd also in /folder0/folder1, which has:

cimport numpy as np     

      # <- first question: is this correct? I assume I need something of this sort, but not sure

      # (I couldn't get it work with any variation - with or without import / cimport numpy in the .pxd - I could think of)

and then for for each function in _shared_ct.pyx:

cdef np.ndarray[np.float64_t, ndim=3] example_func(np.ndarray[np.float64_t, ndim=1], double[:,:,::1], int , int , int [:,::1], int, bint)

i.e. the function with return and argument types, but argument names, body etc removed.

and then for example in extension0.pyx:

from _shared_ct cimport example_func

... but when I run setup.py to build the extensions , one of two things will happen:

a) it will refuse to compile at all; the error message states " _ct_shared.pxd not found"

- this is what happened first

b) if I add e.g. "folder0/folder1" to the include_dirs of module0 in setup.py it appears to compile ok, but then when I import  extension0, it will raise "_shared_ct not found"

I'm  a bit lost what I should do.

Do I need to put _shared_ct as an extension in setup.py as well and compile it first for this to work? That actually would make sense to me, sense, but I never found any mention that this should be necessary, so I'm unsure how to proceed.

Any help appreciated.

(on a sidenote, if I instead

include "_shared_ct.pyx"

in the other modules, all works well, so that's a workaround - but it seems less clean, and apparently can lead to lots of hard to spot problems when recompiling, so I'd rather avoid that if possible.)

Thanks,

David

[David Poensgen]

I am not sure, but I might have gotten one step further by just going through combinations of options:

-> added an extension for _shared_ct to my setup script;

-> added "folder0/folder1" (where .pyx, .pxd and .pyd of _shared_ct are/end up) to the other extensions are

this together still produces "module _shared_ct not found".

-> changed import line to

from ._shared_ct cimport example_func

which now throws a new, heretofore unseen error (yay!), namely

AttributeError: module 'folder0.folder1._shared_ct' has no attribute '__pyx_capi__'

So, now I'll be googling that ... in the meantime, would still be very thankful about any help/hints.

[da-woods]

I haven't had time for a detailed look but:

On 28/04/2022 19:43, David Poensgen wrote:

and then for for each function in _shared_ct.pyx:

cdef np.ndarray[np.float64_t, ndim=3] example_func(np.ndarray[np.float64_t, ndim=1], double[:,:,::1], int , int , int [:,::1], int, bint)

i.e. the function with return and argument types, but argument names, body etc removed.

I'm not sure you want to remove the argument names. Cython will probably interpret `int` as a Python object argument named `int` instead of an integer argument. Not 100% sure about it, but since there's some ambiguity between the C-like and Python-like syntax it's probably better to keep the names. Not sure if that's the sole cause of your issues though.

[David Poensgen]

Thanks for pointing out that possibility, I can try that easily and will do. 

However, I'm a bit puzzled, as the example in the official documentation has the .pxd with just argument type:

https://cython.readthedocs.io/en/latest/src/userguide/sharing_declarations.html#sharing-c-functions

(unfortunately, that example is quite basic; no mentions of project structure, very simple setup.py only, etc.)

[da-woods]

It may still be worth a try but the documentation is probably right. It was a quick guess. I'll try to have a proper look at your initial example in the near future.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/1134e9df-18c2-4cf7-9e4c-b63c2bd3163an%40googlegroups.com.

[da-woods]

Going back through this fully:

> AttributeError: module 'folder0.folder1._shared_ct' has no attribute
'__pyx_capi__'

This seems to be caused by a caching issue. I had to delete the "build"
folder and then rebuild the extensions and then it went away.

My setup.py was:

from distutils.core import setup
from Cython.Build import cythonize
from distutils.extension import Extension
from numpy import get_include

ext_modules = [Extension("folder0.folder1.extension0",
["folder0/folder1/extension0.pyx"],
                                  include_dirs=[get_include()]),
               Extension("folder0.folder1._shared_ct",
["folder0/folder1/_shared_ct.pyx"],
                                  include_dirs=[get_include()])]


setup(
    ext_modules = cythonize(ext_modules)
)


I changed the cimport statement in example0.pyx to:

from folder0.folder1._shared_ct cimport example_func


I had to add __init__.pxd files in folder0 and folder1.


A few more comments on the quoted post below:

On 28/04/2022 16:23, David Poensgen wrote:
> I understand I need a _shared_ct.pxd file that contains the function
> signatures for that. So I have a file, _shared_ct.pxd also in
> /folder0/folder1, which has:
> cimport numpy as np
>       # <- first question: is this correct? I assume I need something
> of this sort, but not sure
>       # (I couldn't get it work with any variation - with or without
> import / cimport numpy in the .pxd - I could think of)

Yes this is correct. You need "cimport numpy" to use the Numpy c
definitions. You should also add np.import_array() in the pyx file (you
can sometimes get away without it, but it's better to have it)

> Do I need to put _shared_ct as an extension in setup.py as well and
> compile it first for this to work? That actually would make sense to
> me, sense, but I never found any mention that this should be
> necessary, so I'm unsure how to proceed.

Yes you do need to do that - see the example setup.py file above

>
>
> (on a sidenote, if I instead
> include "_shared_ct.pyx"
> in the other modules, all works well, so that's a workaround - but it
> seems less clean, and apparently can lead to lots of hard to spot
> problems when recompiling, so I'd rather avoid that if possible.)

This was the (very) old way of doing it.

[David Poensgen]

Thank you so much! 

I got it to work, following the way you laid out. 

- Actually, for me the __init__.pxd files do not even seem to be necessary. (but I have __init__.pys in place, maybe that already does the trick?)

- Also, both from "folder0.folder._shared_ct cimport ..." and "from _shared_ct cimport ..." seem to work fine.

Not actually sure what made the difference (I thought I had tried similar combinations before), but I'm more than happy to just take it. Perhaps the caching issue is a good bet indeed.

(btw, is there a way to tell cython in setup.py to just always do a full recompile? I feel this would have saved me a few headaches, also with other issues. If not, I should just get into the habit of running an according script before setup.) 

A quick follow-up-question: After I cythonize the extensions as discussed above, it should alternatively also be possible to install all extensions from the resulting .c files?

I.e. the interdependency of the .pyx modules is resolved in those without me having to do anything more?

(result of a quick experiment: yes, it works - but it's of course hard for me to test whether that would port well to other systems/situations)

And thank you also for the additional, helpful and clarifying comments.

[da-woods]

On 30/04/2022 20:27, David Poensgen wrote:
> (btw, is there a way to tell cython in setup.py to just always do a
> full recompile? I feel this would have saved me a few headaches, also
> with other issues. If not, I should just get into the habit of running
> an according script before setup.)

Passing --force to setup.py should do it but didn't seem to help in this
case. I'm not completely sure why.

>
> A quick follow-up-question: After I cythonize the extensions as
> discussed above, it should alternatively also be possible to install
> all extensions from the resulting .c files?
> I.e. the interdependency of the .pyx modules is resolved in those
> without me having to do anything more?
> (result of a quick experiment: yes, it works - but it's of course hard
> for me to test whether that would port well to other systems/situations)
>

Yes. The .c files should be independent of the platform you generate
them on. At one point we recommended distributing the .c files as a
forward-compatible distribution method (and it avoids needing Cython as
an install dependency). Python's C API is at a faster rate now so it
isn't so useful for forward compatibility, but it's still an option.

[David Poensgen]

Great, no further questions on this right now, thank you again for your help!

[Stefan Behnel]

David Poensgen schrieb am 29.04.22 um 20:19:

> However, I'm a bit puzzled, as the example in the official documentation
> has the .pxd with just argument type:
> https://cython.readthedocs.io/en/latest/src/userguide/sharing_declarations.html#sharing-c-functions

That's because it's a really trivial example (in which 'x' passes as an
acceptable argument name). We generally should not encourage users to do
this. It's usually safe, but it makes the signature harder to read for
humans (and we write code for humans and not for computers), and it
prevents the usage of keyword arguments. In the example, it's sufficiently
unlikely that anyone would pass 'x' as a keyword argument, but it still
feels ambiguous for me as a human reader to see this function signature.

I'll change it in the documentation. Thanks for pointing this out.

Stefan

[Stefan Behnel]

da-woods schrieb am 30.04.22 um 11:11:

> On 28/04/2022 16:23, David Poensgen wrote:
>> Do I need to put _shared_ct as an extension in setup.py as well and
>> compile it first for this to work? That actually would make sense to me,
>> sense, but I never found any mention that this should be necessary, so
>> I'm unsure how to proceed.
>
> Yes you do need to do that - see the example setup.py file above

Clarification: it needs to be declared as an extension (because that's what
it is). However, the order in which the extensions are built does not
matter. It's enough to have the .pxd files in place for Cython to get all
necessary information from. The .c files are then independent from each
other and can be compiled in any order.

Stefan

[David Poensgen]

Hi Stefan,

thanks for the additional clarifications and making the documentation more clear on this!

If I may make a sugggestion in that regard: I think it might help to make it more clear/prominent in the doc that a separate extension needs to be declared in setup.py. 

This is mentioned, but only under "Sharing Extension Types", so one may miss it easily when looking for functions specifically, as I apparently did at first. 

And perhaps that is obvious to experienced users/those who understand what is going on under the hood; but when I first came across this topic, I think I somehow presumed it would work more like "include" in the sense that the importing .pyx would 'look up' the function body and compile it to .c along its own functions as necessary when it is built. (If that even makes sense...)

Again, thanks to both of you for your help and your work in general!

Best,

David

[David Poensgen]

To anyone who comes across this in the future:

One potential pitfall, as I just realized, has to do with the language level that's passed to cythonize:

Implicit relative imports are legal in python2, but not in 3.

This carries over to where cython will look for *.pxd files.

In particular, for the example I stated above, this:

from _shared_ct import ...

works when the language level is 2, but not when it's set to 3. To have it work with 3 as well, it should be

from ._shared_ct import ...

... I was just testing different ways to configure setup.py and run into the issue why it would sometimes suddenly complain about a missing .pxd, and sometimes compile just fine - I inadvertently was changing the language level, causing this issue to pop up.