Help with cimports in pure python mode

[Ewan Davies]

I have a small project with some Python and some Cython modules and I'm successfully using setuptools to build the Cython modules. The code is mainly Python-like and doesn't have many C-isms, I just want fast loops over simple Python objects so turned to Cython. 

I would like to migrate my .pyx files to .py so that I can enjoy better language server/IDE features, but I'm struggling with cimports. I am successfully using pure Python mode on some Cython files that do not cimport anything, but for the .pyx files I have with a cimport line I cannot seem to persuade the compiler to be nice to me in pure Python mode.

Here's an example. Suppose that mycode.pyx contains the lines

---

from sage.graphs.base.dense_graph cimport DenseGraphBackend

...

@cython.cclass
class LocalViewBackend(DenseGraphBackend):

  ...

---

where DenseGraphBackend is a cdef class defined in (a pxd file in) a dependency: sagemath-standard. Then the mycode.pyx defines its own cdef class (using pure python compatible syntax).

If I change the extension to .py and the line with the cimport to

---

from cython.cimports.sage.graphs.base.dense_graph import DenseGraphBackend

---

then I get compiler errors relating to missing .h files. I have chased down those compiler errors and the culprit seems to be that with the .pyx extension and cimport syntax, the generated .c file contains the following distutils metadata:

---

        "depends": [
            "/Users/ewandavies/.local/share/mamba/envs/sage/lib/python3.12/site-packages/cysignals/macros.h",
            "/Users/ewandavies/.local/share/mamba/envs/sage/lib/python3.12/site-packages/cysignals/struct_signals.h",
            "/Users/ewandavies/.local/share/mamba/envs/sage/lib/python3.12/site-packages/sage/cpython/string_impl.h",
            "/Users/ewandavies/.local/share/mamba/envs/sage/lib/python3.12/site-packages/sage/data_structures/bitset_intrinsics.h"
        ],
        "include_dirs": [
            "/Users/ewandavies/.local/share/mamba/envs/sage/lib/python3.12/site-packages/sage/data_structures",
            "/Users/ewandavies/.local/share/mamba/envs/sage/lib/python3.12/site-packages/cysignals",
            "/Users/ewandavies/.local/share/mamba/envs/sage/lib/python3.12/site-packages/sage/cpython"
        ],
        "libraries": [
            "gmp"
        ],

---

but this is missing from the generated .c file when I use .py. Lo and behold, I get a missing "struct_signals.h" error I compile that version. Have I misunderstood how cimports work in pure Python mode? Is it a Cython bug that changing to pure Python mode somehow causes the Cython compiler to fail to figure out the depends, include_dirs, and libraries for me? I'm aware that I can manually supply those in setup.py, but this isn't necessary for the .pyx version of the module. I have tried putting exactly those depends, include_dirs, and libraries options into setup.py when I declare the Extension, and yes it does fix the compiler error for the pure Python version of the module.

Obviously it's nicer with the .pyx file that the compiler figures out my dependencies for me, but I'd take the hit on that to get linters and better IDE tooling with .py files if I can work around it by manually specifying my own dependencies. But how can I do that portably? What if someone who installed sagemath in a mamba environment with a different path wants to compile? Perhaps there are some environment variables that would help me but I'm a bit stuck on my own.