Importing a cython module from another module

[Nicolas Essis-Breton]

I would like to import the package cythonAnimal into a module located
in the package pythonCaller.
But I keep getting an import error.

My directory is as follow:

sampleProject/
cythonAnimal/
__init__.py
cat.pxd
cat.pyx
dog.pxd
dog.pyx
setup.py
pythonCaller/
__init__.py
outOfPackageCall.py

outOfPackageCall.py:
from cythonAnimal.cat import Cat #Fail here with the error below
from cythonAnimal.dog import Dog

cat = Cat()
dog = Dog()

print cat.name() + ' is friend with ' + cat.friendWith() #should
print 'cat is friend with dog'
print dog.name() + ' is friend with ' + dog.friendWith() #should
print 'dog is friend with cat'

The error is:
Traceback (most recent call last):
File "/sampleProject/pythonCaller/outOfPackageCall.py", line 1,
in <module>
from cythonAnimal.cat import Cat
File "dog.pxd", line 3, in init cat (cat.c:962)
ImportError: No module named dog

The faulty line in question (cat.c:962) is:
__pyx_ptype_3dog_Dog = __Pyx_ImportType("dog", "Dog",
sizeof(struct __pyx_obj_3dog_Dog), 1); if (unlikely(!
__pyx_ptype_3dog_Dog)) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 3;
__pyx_clineno = __LINE__; goto __pyx_L1_error;}

My other file are as follows:

cat.pxd:
from dog cimport Dog
cdef class Cat:
cpdef name(self)
cpdef friendWith(self)

cat.pyx:
cdef class Cat:
cpdef name(self):
return 'cat'
cpdef friendWith(self):
cdef Dog dog = Dog()
return dog.name()

dog.pxd:
from cat cimport Cat
cdef class Dog:
cpdef name(self)
cpdef friendWith(self)

dog.pyx:
cdef class Dog:
cpdef name(self):
return 'dog'
cpdef friendWith(self):
cdef Cat cat = Cat()
return cat.name()

setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import os

packageDir = os.path.dirname(__file__)
includedDir = [packageDir]
os.chdir(packageDir)

ext_modules = [
Extension("cat", ["cat.pyx"], include_dirs=includedDir),
Extension("dog", ["dog.pyx"], include_dirs=includedDir)
]

setup(
name='cythonAnimal',
cmdclass={'build_ext': build_ext},
include_dirs=includedDir,
ext_modules=ext_modules,
script_args=['build_ext'],
options={'build_ext':{'inplace':True, 'force':True}}
) #I run my setup.py directly from the interpreter with:
reload(cythonAnimal.setup)

[Nicolas Essis-Breton]

I found cat.pyx and dog.pyx create a circular import.
I tried to remove it, by making cat.pyx independent and dog.pyx dependent of cat, but I still get the same problem when I try to import dog.pyx in another module.

I saw somewhere that setup.py should be in the root directory, not directly in the package directory. This didn't help either.

I tried in dog.pxd, without success, to do

cimport cat

from cat cimport Cat

I have attached the code from the original question.
I guess this is a newbie issue. I tried hard to find a solution but nothing worked.
Any help appreciated.

[Nicolas Essis-Breton]

I found a solution to this problem.
I will post a working sample project soon.
Briefly, the directory structure is incorrect for setup to effectively create the package.
Also the include_dirs in my setup.py is incorrect.

[Bradley Froehle]

You'll need to use absolute cimports in your pxd files. That is, change the first line of cat.pxd to

    from cythonAnimal.dog cimport Dog

[Nicolas Essis-Breton]

I have attached a working multi package project.

The project illustrate:
-how to layout the directory structure
-how to write a setup file where each pyx is specify by hand
-how to write a setup file where each pyx is detected automatically

Thanks for the pointer Bradley.

I think this simple example would have help me a lot when I was reading http://wiki.cython.org/PackageHierarchy.
I tried to add it to the wiki page, but I don't have permission. Can someone do it?

[Rui Lopes]

Hi guys,

I have a library that is quite big already, and has a few dependencies like bitarray, for instance.

I am trying to Cythonize the python files in order to test the approach before refactoring and adding .pdx and .pyx files. 

I am having two issues. First I tried manually, Cython creates .so for each file with success, but I cannot import/link bitarray (it is a C extension itself, the module has a __init__.py and a _bitarray.so).

After some googling  I found the PackageHierarchy tutorial and this thread. But using these methods I cannot import my inner modules, only the root (I am using the fully qualified names, I verified that the script is building them correctly, and I have the '.' in the includedirs).

Is it even possible to do it like this (without pyx and pdx)? How can I try cython without massively refactoring my code?

You can have a glimpse of the library here, I am trying to cythonize the 'code' module.

Thank you in advance,

Rui

[Stefan Behnel]

Nicolas Essis-Breton schrieb am 10.05.2012 um 15:37:
> I have attached a working multi package project.
>
> The project illustrate:
> -how to layout the directory structure
> -how to write a setup file where each pyx is specify by hand
> -how to write a setup file where each pyx is detected automatically

The package hierarchy is the same as in Python. If you put your source
files in the spot where the compiled module should live (as you would for
.py files), and then build everything from a setup.py in the root directory
(as you would for Python packages), it should just work.

> I think this simple example would have help me a lot when I was reading
> http://wiki.cython.org/PackageHierarchy.
> I tried to add it to the wiki page, but I don't have permission.

That Wiki page was moved here:

https://github.com/cython/cython/wiki/PackageHierarchy

Improvements are welcome.

Stefan

[Rui Lopes]

Hi Stefan

I was missing the __init__.py in the .so hierarchy. I made a reduced version of the library and got it running, half an hour ago.

I am getting around 10% improvement in performance. Is not much, even given that bitarray was already a C extension, I was expecting a bit more.

Any suggestions regarding how to extract maximum performance gains?

Thanks

Rui

[Robert Bradshaw]

On Tue, Jun 16, 2015 at 8:05 AM, Rui Lopes <rui.loure...@gmail.com> wrote:
> Hi Stefan
>
> I was missing the __init__.py in the .so hierarchy.

Did you not have an __init__.py in your original Python hiearchy?

> I made a reduced version
> of the library and got it running, half an hour ago.
> I am getting around 10% improvement in performance. Is not much, even given
> that bitarray was already a C extension, I was expecting a bit more.
>
> Any suggestions regarding how to extract maximum performance gains?

That really depends on what you're doing. If you're just taking Python
objects and passing them to C for the heavy lifting, there's not
really any time to cut out. You could probably get gains eliminating
the bitarray -> _bitarray calls and using the library directly in
bitarray.pyx.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[klo...@gmail.com]

Hi,

Just checking in whether there is a way to compile a large, single extension, where extension types are "namespaced" in submodules.

From my research, the best way of creating a large, single extension is with the include directive, as illustrated here:

https://github.com/chadmv/plow/tree/master/lib/python/src

However, using this approach, all types end up in the top-level module. From what I can gather, my understanding is that this is currently the only option, but maybe things have changed.

Cheers,
kloffy

[Robert Bradshaw]

On Fri, Jun 26, 2015 at 10:38 AM, <klo...@gmail.com> wrote:
> Hi,
>
> Just checking in whether there is a way to compile a large, single
> extension, where extension types are "namespaced" in submodules.
>
> From my research, the best way of creating a large, single extension is with
> the include directive, as illustrated here:
>
> https://github.com/chadmv/plow/tree/master/lib/python/src
>
> However, using this approach, all types end up in the top-level module.

Yes, include is literally copy-paste, like the C include directive.

> From
> what I can gather, my understanding is that this is currently the only
> option, but maybe things have changed.

You could link things together and manually call the import modules
yourself, but I think that'd be tricky. Is there a reason you need to
do this? Python doesn't easily let you create multiple modules with a
single .py[c] file.

[klo...@gmail.com]

Hi, thank you for the quick response! I am wrapping a C++ library and without namespaces there are naming collisions. I am aware that I can simply change the names in my python extension, but it would be more "natural" to have the namespaces correspond to submodules. Compiling the library into separate extensions is not an options since there is shared internal state.

Cheers,
kloffy

[Stefan Behnel]

Hi,

please don't top-post.

klo...@gmail.com schrieb am 27.06.2015 um 08:43:
> On Saturday, June 27, 2015 at 4:13:05 PM UTC+10, Robert Bradshaw wrote:

>> On Fri, Jun 26, 2015 at 10:38 AM, <klo...@gmail.com <javascript:>>
>> wrote:
>>> Just checking in whether there is a way to compile a large, single
>>> extension, where extension types are "namespaced" in submodules.
>>>
>>> From my research, the best way of creating a large, single extension is
>>> with the include directive, as illustrated here:
>>>
>>> https://github.com/chadmv/plow/tree/master/lib/python/src
>>>
>>> However, using this approach, all types end up in the top-level module.
>>
>> Yes, include is literally copy-paste, like the C include directive.
>>
>>> From
>>> what I can gather, my understanding is that this is currently the only
>>> option, but maybe things have changed.
>>
>> You could link things together and manually call the import modules
>> yourself, but I think that'd be tricky. Is there a reason you need to
>> do this? Python doesn't easily let you create multiple modules with a
>> single .py[c] file.
>

> Hi, thank you for the quick response! I am wrapping a C++ library and
> without namespaces there are naming collisions. I am aware that I can
> simply change the names in my python extension, but it would be more
> "natural" to have the namespaces correspond to submodules. Compiling the
> library into separate extensions is not an options since there is shared
> internal state.

The normal C/C++ way to deal with "shared internal state" is to link
dynamically against an external library.

Stefan

[klo...@gmail.com]

Apologies, what do you mean by top-post? (I'm a bit of a newb when it comes to Google Groups.)

Yes, having an external library would solve the "shared internal state" problem. However, I was trying to avoid the approach due to problems like this:

http://stackoverflow.com/questions/5661738/how-can-i-use-standard-library-stl-classes-in-my-dll-interface-or-abi

Cheers,
kloffy

[Robert Bradshaw]

On Sat, Jun 27, 2015 at 12:02 AM, <klo...@gmail.com> wrote:
> Apologies, what do you mean by top-post? (I'm a bit of a newb when it comes
> to Google Groups.)

https://www.google.com/search?q=top-posting

> Yes, having an external library would solve the "shared internal state"
> problem.

Can you not encapsulate your state in one module and reference it
everywhere else? If the library you're trying to wrap does not allow
for that, I'd file a bug.

> However, I was trying to avoid the approach due to problems like
> this:
>
> http://stackoverflow.com/questions/5661738/how-can-i-use-standard-library-stl-classes-in-my-dll-interface-or-abi

As long as you're building with the same compiler, this should not be an issue.

[klo...@gmail.com]

On Saturday, June 27, 2015 at 5:35:54 PM UTC+10, Robert Bradshaw wrote:

On Sat, Jun 27, 2015 at 12:02 AM,  <klo...@gmail.com> wrote:
> Apologies, what do you mean by top-post? (I'm a bit of a newb when it comes
> to Google Groups.)

https://www.google.com/search?q=top-posting

 

Gotcha. :)
 

> Yes, having an external library would solve the "shared internal state"
> problem.

Can you not encapsulate your state in one module and reference it
everywhere else? If the library you're trying to wrap does not allow
for that, I'd file a bug.

> However, I was trying to avoid the approach due to problems like
> this:
>
> http://stackoverflow.com/questions/5661738/how-can-i-use-standard-library-stl-classes-in-my-dll-interface-or-abi

As long as you're building with the same compiler, this should not be an issue.

Just to make sure that I understand correctly: This means I would have one shared library (.so/.dll) and multiple cython extension modules, which would dynamically link to it. In order to safely use stl classes across interface boundaries, the shared library and extension modules would have to be built with the same compiler. I actually experimented with this kind of setup before I switched to static linking. I suppose I could revert to it.