Extension module in different folder than .pxd

[Paul Londino]

Hi,

I am creating a Cython module, but I have a growing number of pyx and pxd files. I want to place the .pyx and .pxd files in a separate directory and compile them into extension modules that live in a parent directory. For example, a module laid out as follows:

package_a/

  __init__.py

  pure_python.py

  cython_files/

    rect.pxd

    rect.pyx

    cy_module.pyx

I want to create the following extension modules:

ext_modules = [

    Extension('package_a.rect', sources=['package_a/cython_files/rect.pyx']),

    Extension('package_a.cy_module', sources=['package_a/cython_files/cy_module.pyx'])

]

However, this fails on import of package_a.cy_module which cimports rect.pxd:

>>> import package_a.cy_module

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "package_a\cython_files\rect.pxd", line 10, in init package_a.cython_files.cy_module(package_a/cython_files/cy_module.cpp:1070)

    cdef class PyRectangle:

ImportError: No module named rect

 I tried various combinations of relative c imports and absolute c imports, but the only thing that worked was changing the extension modules to:

ext_modules = [

    Extension('package_a.cython_files.rect', sources=['package_a/cython_files/rect.pyx']),

    Extension('package_a.cython_files.cy_module', sources=['package_a/cython_files/cymodule.pyx'])

]

Is there any way to do what I'm trying to do and still cimport the .pxd of another module?

Thanks,

Paul

[Robert Bradshaw]

On Wed, May 25, 2016 at 6:59 PM, Paul Londino <lond...@gmail.com> wrote:
> Hi,
>
> I am creating a Cython module, but I have a growing number of pyx and pxd
> files. I want to place the .pyx and .pxd files in a separate directory and
> compile them into extension modules that live in a parent directory.

This is not supported--Cython expects your .pyx and .pxd files to be
organized out as if they were .py files. Why do you want to do this?

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Paul Londino]

Sorry, I think the title I wrote was misleading. I really want to separate out from the pure python code (but keep together) all of the .pyx and .pxd into a subfolder (either as part of a subpackage or even outside the package in a "cython_files" folder or something like that) but then have the module built by Cython to be in the same package.

So like for pure Python, where the folder structure on disk doesn't have to mimic the package layout, I want to have packages like 'package_a.cy_module' even though the source (pxd and pyx files) love in a separate directory like 'package_a/cython_files/cymodule.pyx'. 

I had this working fine until I wanted to import something into a .pyx file from the .pxd of another Cython module, but something about how the imports work seems to prevent it being able to find what's imported from the other Cython module. I tried:

from package_a.cython_files cimport rect

which results in the runtime import error and

from package_a cimport rect

which results in a Cython compile error (about not being able to find the .pxd)

So I'm curious if this should be possible and I'm just missing something, or if it's not possible.

[Robert Bradshaw]

On Thu, May 26, 2016 at 12:30 PM, Paul Londino <lond...@gmail.com> wrote:
> Sorry, I think the title I wrote was misleading. I really want to separate
> out from the pure python code (but keep together) all of the .pyx and .pxd
> into a subfolder (either as part of a subpackage or even outside the package
> in a "cython_files" folder or something like that) but then have the module
> built by Cython to be in the same package.
>
> So like for pure Python, where the folder structure on disk doesn't have to
> mimic the package layout,

In pure Python, the folder structure on the disk does mimic (in fact
it determines) the package layout (along with the __init__.py files).
The same is true for Cython.

> I want to have packages like 'package_a.cy_module'
> even though the source (pxd and pyx files) love in a separate directory like
> 'package_a/cython_files/cymodule.pyx'.
>
> I had this working fine until I wanted to import something into a .pyx file
> from the .pxd of another Cython module, but something about how the imports
> work seems to prevent it being able to find what's imported from the other
> Cython module. I tried:
>
> from package_a.cython_files cimport rect
>
> which results in the runtime import error and
>
> from package_a cimport rect
>
> which results in a Cython compile error (about not being able to find the
> .pxd)
>
> So I'm curious if this should be possible and I'm just missing something, or
> if it's not possible.

This is not possible as cimport must do both a static import and a
runtime import against the same path.

If you want your cython modules next to your python modules, put your
.pyx files next to your .py files. Alternatively you can put them in a
subpackage (and import them from this subpackage). If you really want,
you could create .py files that import * from your .pyx files that
live in a subpackage, but that's just extra overhead and cimports
would still have to come from the proper subpackage.

It's a feature, not a bug, that you can put .py and .pyx files in the
same directory. This also means your users won't have to take any
action (or even know) if you decide to cythonize (or uncythonize) a
given module.

Your setup.py listing a module as "package_a.cy_module" while its
source is "package_a/cython_files/cy_module.pyx" is causing the
discrepancy between imports and cimports here--being consistent will
save you a lot of headaches. (I take it back, it might be possible if
you manually muck around with sys.modules and import hooks to hide the
fact that these modules are in a subpackage, but I wouldn't advise
that).

[Paul Londino]

Thanks for taking the time for a comprehensive responce.  What I meant about the pure Python not having to mimic the filesystem, I meant that the package as it is being developed doesn't have to mimic the filesystem and that you can use arguments to distutils to describe how what's on disk maps to the package structure:

https://docs.python.org/2/distutils/setupscript.html#listing-whole-packages

Of course once the package is installed they will live together in a folder structure that matches the package hierarchy. I thought it would be possible to do the same thing with Cython by changing the output package argument to the extension e.g.

Extension('package_a.rect', sources=['package_a/cython_files/rect.pyx']),

which does work until .pxd's are involved, which seems to be as you say because it needs the static lookup during compilation to be in the original folder but the runtime import needs to match the new package structure.

This is mostly an organization issue at the development stage, so definitely agree mucking around in module imports is not the way to go.  Thanks again.