Odd behavior: pxd not linked to corresponding pyx without self-import in pyx

[Ethan Craigo]

Hi there,

I'm currently running a project with Python 3.13.0 and Cython 3.0.11 (endeavoring to speed up some bottlenecks on a very computationally intensive Python project using Cython). I'm very new to Cython and just figuring out how it works.

I'm not sure if pyximport is the root cause of the problem, if Cython generally is, or if it's just my own ignorance (almost certainly my own ignorance), but something seems odd in linking corresponding pxd and pyx files.

Important to note: I'm working out of a virtualenv defined through virtualenvwrapper, which adds my project root folder ".../project_root" to sys.path in Python.

The main project script I'm running uses pyximport at the top of the file to do an in-place install of all my Cython scripts like so:

project_root/run/run.py:

...

import pyximport
pyximport.install(inplace=True, language_level=3)

...

from project_root.foo.cython import caller

...

Suppose now that I have a folder "project_root/foo/test/cython" with contents like so:

project_root/foo/cython/test.pxd:

cdef int n

project_root/foo/cython/test.pyx, version 1:

cdef int n = 4

project_root/foo/cython/caller.pyx:

from project_root.foo.cython.test cimport n

# <do something with n>

...

Running run.py now eventually returns the following error:

AttributeError: module 'project_root.foo.cython.test' has no attribute '__pyx_capi__'

Looking up the cause of this led me to pages online which warned me that this error would appear if I didn't have corresponding .pxd, .pyx, and .so files in exactly the same folder with exactly the same names. This is already the case, unfortunately. I should also mention that there are also absolutely no __init__.py or __init__.pxd files anywhere throughout my project folders, as I know those can confuse Cython too.

After spending probably six solid hours trying to figure out what was going on here, I identified a workaround. Let's modify test.pyx like so:

project_root/foo/cython/test.pyx, version 2:

from project_root.foo.cython.test cimport n  # new line

cdef int n = 4

Suddenly, everything works. The pxd file is properly linked to the pyx file, as whatever needs to generate __pyx_capi__ finally does its job. And thankfully, caller.pyx now correctly recognizes that project_root.foo.cython.test.n is equal to 4.

Now, given that I've encountered no trouble with this workaround so far, I could just proceed onward with this odd syntax. But I really shouldn't have to make test.pyx "cimport" its own .pxd file, should I? According to the documentation I've read, it ought to be unnecessary.

I was thinking that maybe I had reached the limits of pyximport.install's capabilities, but then I realized that there's already a difference in generated C code that becomes quite apparent when I run "cython -a project_root/foo/cython/test.pyx" on both versions of test.pyx I've just shown above – no pyximport involved.

In the outputted html for each version, I can expand the "cdef int n = 4" line and see the generated C respectively:

Version 1:

...

+1: cdef int n = 4
  __pyx_v_4test_n = 4;

...

Version 2:

...

+1: cdef int n = 4
  __pyx_v_8project_root_11foo_6cython_4test_n = 4;

...

It's clear that Version 2's "from project_root.foo.cython.test cimport n" line is causing project path prefixes to work as they should, and Version 1's generated C probably shows the root cause of the cryptic missing-__pyx_capi__ error. Something is going wrong in generating the paths of declarations relative to my project root folder by default, and this is being gently corrected by the explicit self-import. It may be unreasonable for me to expect that the pyx/pxd files "just know" the absolute paths with which I want to prefix them, but I see no clear and easy place in pyximport or elsewhere to override this.

I usually pride myself on being able to figure out gnarly issues like this on my own, but I'm at about the end of my rope here. Does anyone know a way to deal with this odd behavior without adding a bunch of boilerplate/overhead for each new .pyx file I write?

All the best,

Ethan

[Stefan Behnel]

Hi,

Ethan Craigo schrieb am 03.12.24 um 18:14:
> *project_root/run/run.py:*

> ...
> import pyximport
> pyximport.install(inplace=True, language_level=3)
> ...
> from project_root.foo.cython import caller
> ...
>
> Suppose now that I have a folder "project_root/foo/test/cython" with
> contents like so:
>

> *project_root/foo/cython/test.pxd:*
> cdef int n
>
> *project_root/foo/cython/test.pyx, version 1:*

> cdef int n = 4
>

> *project_root/foo/cython/caller.pyx:*
> from project_root.foo.cython.test cimport n

This implies that you have a Python package "project_root.foo.cython"

> Running run.py now eventually returns the following error:
> AttributeError: module 'project_root.foo.cython.test' has no attribute
> '__pyx_capi__'
>
> Looking up the cause of this led me to pages online which warned me that
> this error would appear if I didn't have corresponding .pxd, .pyx, and .so

> files in *exactly* the same folder with *exactly *the same names. This is

> already the case, unfortunately. I should also mention that there are also
> absolutely no __init__.py or __init__.pxd files anywhere throughout my
> project folders, as I know those can confuse Cython too.

That's most likely the issue. If you want to create a Python package, use
"__init__.py" files to mark the directories as package directories. Cython
is in no way different from Python here.

Stefan

[Ethan Craigo]

Hi Stefan; thank you for the response!

> This implies that you have a Python package "project_root.foo.cython"

Yes, but it's a PEP 420 implicit namespace package, with no __init__.py anywhere. I like to keep my project folders small and clean of unnecessary boilerplate wherever possible, so I don't write __init__.py files in my Python 3.3+ codebases unless atypical circumstances require them.

> That's most likely the issue. If you want to create a Python package, use

> "__init__.py" files to mark the directories as package directories. Cython
> is in no way different from Python here.

I would beg to differ about whether Cython is different from Python here. Python has no problem running on my project exactly as it is, and fully recognizes imports from any "project_root.foo.bar" subdirectory without the need for writing __init__.py in ...foo/bar/. Cython, on the other hand, looks to be having a difficult time doing so.

I'm happy to keep on with my strange workaround for now, if indeed this is the problem. But I'd read that Cython was theoretically compatible with implicit namespace packages, since https://github.com/cython/cython/issues/2918 has been completed – so I'd expected things to work.

If indeed Cython is supposed to support implicit namespace packages, could this be a bug?

Best,

Ethan