pxd import problem when modules added to package

[Alex Meakins]

Hello I'm wondering if anyone can help me, I've been fighting a pxd import battle for a good 4 hours and I've not made any progress.

I am developing a tightly knitted set of fast vector routines for a raytracer I'm working on and have hit a stumbling block when I tried to refactor my code to split a set of interdependent objects into separate files (the original was getting too long). I set up pxd files for the objects, the compiler appears to find the files, but when I test the objects I keep getting a module not found error.

The odd thing is that all the code works correctly if I place the split files in the same directory as the setup script ie not under a package folder. However, as soon as I place it under a package folder the code breaks! Here is a toy version of the code I've written:

a.pxd:

from b cimport B

cdef class A:

 

   pass

 a.pyx:

cdef class A:

    def __init__(self):

        print("A__init__()")

   def test(self):

       cdef B r

       r = B.__new__(B)

       return r

b.pxd:

from a cimport A

cdef class B(A):

   pass

b.pyd:

cdef class B(A):

def __init__(self):

   print("B.__init__()")

If I place these files in a folder along with this setup.py file, everything works fine:

setup.py (flat version):

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

extensions = [
   Extension("a", ["a.pyx"]),
   Extension("b", ["b.pyx"]),
   ]

setup(
   ext_modules = cythonize(extensions)
)

 Compiles fine, importing a.A and calling test() prodices a B() object. Great.

But if I now create a package folder testpkg and place a.pxd, a.pyx, b.pxd and b.pyx inside it and alter the setup.py to poitn to the moved files:

setup.py (package version):

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

extensions = [
   Extension("testpkg.a", ["testpkg/a.pyx"]),
   Extension("testpkg.b", ["testpkg/b.pyx"]),
   ]

setup(
   ext_modules = cythonize(extensions)
)

The code compiles ok, but I see the following warning appears:

missing cimport in module 'b': testpkg/a.pxd

missing cimport in module 'a': testpkg/b.pxd

If I then try to import the objects into python I get the following at the end of the Traceback:

/home/##############/cimporttest/inpackage/b.pxd in init a (testpkg/a.c:1001)()

ImportError: No module named 'b'

It looks like a problem with the pxd import, but then how can it compile if it did not find the pxds in the first place. I'm mystified.... I've tried forcing the cython comiler to search for the pxds in the testpkg folder by setting include_dirs paths in the extension definitions but nothing works.

I'm using the latest release of cython 19.2.

This is currently a complete nightmare. Any help your be greatly appreciated!

 

[Stefan Behnel]

Alex Meakins, 07.01.2014 20:47:

> Hello I'm wondering if anyone can help me, I've been fighting a pxd import
> battle for a good 4 hours and I've not made any progress.
>
> I am developing a tightly knitted set of fast vector routines for a
> raytracer I'm working on and have hit a stumbling block when I tried to
> refactor my code to split a set of interdependent objects into separate
> files (the original was getting too long). I set up pxd files for the
> objects, the compiler appears to find the files, but when I test the
> objects I keep getting a module not found error.
>
> The odd thing is that all the code works correctly if I place the split
> files in the same directory as the setup script ie not under a package
> folder. However, as soon as I place it under a package folder the code
> breaks!

Are you sure it's a package folder? I.e. does it contain an __init__.py file?

Stefan

[Alex Meakins]

Well now this is fun, I get to answer my own question.

OK... feel like a twit time.

mistake 1...forgot the __init__.py in the testpkg directory, don't rush your test cases (arrgh... I want my hours back)

mistake 2: You MUST use absolute package paths in cimports. ie:

    from testpkg.a import A, 

When I fixed the above items it started working fine. mistake 1 was a doozy (newbie idiotic moronic rushed error... learn from my public humiliation! :) ).

Adding absolute cimports in my real code worked fine.

A package skeleton would be a great thing to add to the cython docs!

FYI in the full code the import for the derived class had to be moved to the last line of the parent class' pxd file to ensure the parent class was fully defined. Cyclic dependencies are a bitch.

[Alex Meakins]

Thanks Stefan, I just noticed your reply. Must have been typing mine when you sent it.