Cythonizing and cimporting modules in sub-directories on Windows and Linux (Error)

[Source]

Hey. I'm trying to chop my code into modules, compile them as shared libraries, cimport them from my main file, but also be able to use them as standalone modules, as well as being able to use the C++ source code directly in regular C++ projects in the future.

The main problem I'm having is that if I don't have a __init__.pxd or py file in the 'modules' subfolder then none of the modules/names.pxd will be found on Windows, but if I have a __init__.pxd or .py file then any of the modules that depends on (cimporting) any other module will result in a 'module.pxd' not found error caused by the cimport statement on Linux (so cythonize finds the module/main.pxd but main.pxd tries to cimport other.pxd which fails because it can't be found somehow due to the __init__.pxd/.py file on Linux only, remove the file and it works fine on Linux but not on Windows).

In case it matters all of these modules are just wrapping C++ files using extern from.

The directory structure is basically:
/main.pyx

/modules/*.pyx/pxd -> (cimport...) -> extern from cpp/...

/modules/cpp/*.cpp

Version: 3.0.0a10

I also have some more minor bug reports if anyone's interested.

Do I report these as separate PRs as before with test files or can I just inline the code in the PR?

I don't have a lot of experience with using git for any other purpose than VC.

[Source]

I should rephrase the main problem because I missed a part:

So on Linux cythonizing modules in a subfolder that externs from CPP code and cimports each other works fine unless there's a __init__.pxd/py file in the subfolder, in which case I get a 'other.pxd' not found error when one module cimports another, say "main" cimports "other".

On Windows cythonizing the same modules immediately fails if there's no __init__.pxd/py file, the module.pxd files are simply not found prior to evaluating any cimport statements.

Otherwise (this part I forgot to mention) cythonizing will initially find the module.pxd files, but will still fail when one module cimports another.

So for now it seems like I have no choice if I want to compile these on Windows but to avoid having any of the modules cimport each other. and will instead have to combine modules that depend on others (A depends on B, combine A and B)?

I realize maybe it's bad practice in general to have a module or shared library depend on other modules or shared libraries, but the smaller or more fragmented modules are, the less compilation time it should take in the case that I only change and recompile a single module, at least that was the idea, I'm sure it probably depends on the size of the library as well.

(Still after combining modules so that there's no more cimports between modules)

I'm also having some issues with what extern paths to use regardless of OS.

For cythonize to compile the modules both pyx and pxd paths must be "cpp/file.cpp".

However for the compilation of the main program cimporting these the pyx path must be "cpp/file.cpp" and pxd path "modules/cpp/file.cpp".

At least that's the current standing as I haven't found a solution.

If I use "modules/cpp/file.cpp" for both pyx and pxd then neither the modules nor the main program will compile and so on.

In either case of an error the error is an #include failing with "no such file or directory".

[Source]

Sorry, forgot to mention that the __init__.pxd thing seemed to have been solved somehow, maybe by keeping pyx and pxd paths separate or by no longer having modules cimport any other module, I'm not sure.

On Tuesday, March 29, 2022 at 4:07:57 AM UTC-2 Source wrote: