Structuring a large Cython wrapping C++ project with multiple pyx files

[Sandra Hicks]

Hello community,
I have encountered another problem wrapping a C++ library with Cython 0.24.2.
The Library consists of various classes which I want to wrap with Cython. Therefore I wanted to put each class into one pyx file such that I have a structure of:
ClassA.pyx cppClassA.pxd
ClassB.pyx cppClassB.pxd
...
With the pyx file containing the Wrapper class and the pxd file containing the C++ interface. Those Classes also use each other to wrap method returns.

All of those Classes should be compiled to one .so file such that I have a module that wraps the library. I have created a MyLibrary.pyx that includes all of my ClassX.pyx files. My setup.py creates an Extension with MyLibrary.pyx and might also list all of my ClassX.pyx files as source. It does not seem to matter, as both possibilities produce an MyLibrary.so, but if I try to import MyLibrary in python or sage, it throws an error that "No Module named cppClassA". My setup.py links with all required dependencies.

After looking at the generated code the error message points to: in the produced MyLibrary.cpp, it fails where:
/*--- Type import code ---*/
  __pyx_ptype_17cppClassA_ClassA = __Pyx_ImportType("cppClassA", "ClassA", sizeof(struct __pyx_obj_17cppClassA_ClassA), 1); if (unlikely(!__pyx_ptype_17cppClassA_ClassA)) __PYX_ERR(5, 18, __pyx_L1_error)

It fails with the first class that is imported in the "Type import code" section of the generated Code.
Right now all of my files are placed in the same directory. (ClassX.pyx, cppClassX.pxd, MyLibrary.py, setup.py)

What am I doing wrong? Do I need to structure my code differently (I have read about using pxi files), are there more/other possibilities with a newer Cython version? Do I need an explicit __init__.py file and if I do, what does it contain? I have been searching Mailing lists and websites but I cannot seem to find a common opinion on how to do this properly.

[Ayaz Salikhov]

There's an ugly way to make imports work. You need to use PySys_SetPath insider your C++ code. It's well described here: http://stackoverflow.com/questions/39115794/run-python-in-c.

I know that's not very beautiful, but it works. Actually, I have a related question (I need to divide my python code in another file). Could you help me with that, please? It seems from your question that you may know how to do such a thing. https://groups.google.com/forum/#!topic/cython-users/FyCmPI1FWiM

вторник, 13 декабря 2016 г., 16:20:50 UTC+3 пользователь Sandra Hicks написал:

[Chris Barker]

On Tue, Dec 13, 2016 at 5:20 AM, Sandra Hicks <sh.sand...@gmail.com> wrote:

I have encountered another problem wrapping a C++ library with Cython 0.24.2.
The Library consists of various classes which I want to wrap with Cython. Therefore I wanted to put each class into one pyx file such that I have a structure of:
ClassA.pyx cppClassA.pxd
ClassB.pyx cppClassB.pxd
...
With the pyx file containing the Wrapper class and the pxd file containing the C++ interface. Those Classes also use each other to wrap method returns.

All of those Classes should be compiled to one .so file such that I have a module that wraps the library. I have created a MyLibrary.pyx that includes all of my ClassX.pyx files. My setup.py creates an Extension with MyLibrary.pyx and might also list all of my ClassX.pyx files as source. It does not seem to matter, as both possibilities produce an MyLibrary.so, but if I try to import MyLibrary in python or sage, it throws an error that "No Module named cppClassA". My setup.py links with all required dependencies.

IIUC, "Include" in Cython essentially injects teh contents of the file inot that place. So logically, you should hve one big module with all those classes defined. How are you importing them? I would expect that:

from MyLibrary.ClassA import ClassA

would fail as you describe, but:

from MyLibrary import ClassA

should work.

can you import MyLIbrary at all? if so , what names are defined in it?

import MyLibrary

dir(MyLibrary)

It is very possible in Python to have a package with sub-modules in a single extension file (.so) -- but AFAIK, Cython does not provide an easy way to do that.

And it's less conventional in Python to have one class per module anyway -- so while I can see why you'd want to keep your pyx files smaller, is this so huge that it shouldn't be in one namespace?

BTW -- another option is to have a bunch of *.so files -- one for each module, and a simple __init__.py that imports them all.

This all gets delivered as a single package to the user, so really not any harder to deal with.

What am I doing wrong? Do I need to structure my code differently (I have read about using pxi files),

pxi is about sharing declarations, and I think depreciated anyway.

 

are there more/other possibilities with a newer Cython version? Do I need an explicit __init__.py file and if I do, what does it contain?

only if you compile each pyx as a separate module -- then you can use a __init__.py to bring them all into one namespace.

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Robert Bradshaw]

+1, this is the simplest and supported way to do things. Each module
is a single .so file (1:1 with .pyx files, just like Python modules
are 1:1 with .py files), and you can provide an __init__ file that
imports them all into the same namespace.

[Sandra Hicks]

Thanks for the help. I have now taken all of my classes into one pyx file and encapsulated all declarations in one pxd file and it works. As my project is not overwhelmingly large yet, this can still be done. In the future I will also experiment with more than one module and an __init__.py. How would you import this in sage?