Embedding Cython in C++: Epic Fail
seventhunders
working examples so far. I would like to embed python via cython in
some C/C++ code. Thus I'd like to make my cython function calls from C
++, the opposite of what's usually done. The consumer of my code is
not flexible in this due to the real time requirements of the C++
code. Furthermore I have no control over the main function. I can
not use the cython generated main() via the --embed flag
After many failures due to segfaults whenever I try to call into
cython, I figured I was not initalizing something correctly in my
test main function, so I decided to let cython generate main for me.
Here is my test cython code. I am on a Red Hat enterprise linux box.
----------pyembed.pyx--------------------------------
import numpy as np
cimport numpy as np
import sys
import pylab as lab
cdef public void plotxy(float *x, float *y, int len) :
""" Take some simple float arrays and plot them via pylab """
cdef np.ndarray [np.float32_t, ndim=1] xarr = np.empty((len),
dtype=np.float32)
cdef np.ndarray [np.float32_t, ndim=1] yarr = np.empty((len),
dtype=np.float32)
# a probably unnecessary copy to the numpy arrays
cdef int k
for k in range(len) :
xarr[k] = x[k]
yarr[k] = y[k]
# plot the results
fig1 = lab.figure()
lab.plot(xarr, yarr)
lab.grid(True)
fig1.draw()
cdef int len = 30
cdef float xin[30]
cdef float yin[30]
cdef int kk
for kk in range(len) :
xin[kk] = <float> kk
yin[kk] = <float> xin[kk]
plotxy(xin, yin, len)
print "All Done"
-------------------------------------------
I compile the code via these commands:
cython --cplus --embed pyembed.pyx
# create a test executable
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -
Wstrict-prototypes -fPIC -I. -I/usr/local/include/python2.6 -I/usr/
local/lib/python2.6/site-packages/numpy/core/include -I/usr/local/
cuda/include -I/usr/local/cudasdk/common/inc -I/usr/local/include/
python2.6 pyembed.cpp -o embed.run -L. -L/usr/local/cuda/lib64 -L/
usr/local/cudasdk/lib -L/usr/lib64 -L/usr/local/lib/python2.6/site-
packages/numpy/core/lib -lutil -lgpumat -lcuda -lcublas -lcudart -
lcufft -lm -lnpymath -lpython2.6
This actually compiles, though there are many warnings. When I run
it, I get this:
./embed.run
Traceback (most recent call last):
File "numpy.pxd", line 149, in init pyembed (pyembed.cpp:3238)
File "/usr/local/lib/python2.6/site-packages/numpy/__init__.py",
line 132, in <module>
import add_newdocs
File "/usr/local/lib/python2.6/site-packages/numpy/add_newdocs.py",
line 9, in <module>
from lib import add_newdoc
File "/usr/local/lib/python2.6/site-packages/numpy/lib/__init__.py",
line 4, in <module>
from type_check import *
File "/usr/local/lib/python2.6/site-packages/numpy/lib/
type_check.py", line 8, in <module>
import numpy.core.numeric as _nx
File "/usr/local/lib/python2.6/site-packages/numpy/core/
__init__.py", line 5, in <module>
import multiarray
ImportError: /usr/local/lib/python2.6/site-packages/numpy/core/
multiarray.so: undefined symbol: PyType_GenericNew
So numpy is not picking up some runtime library that is essential to
system operation. The problem here is that what I actually need to
link against is probably buried somewhere in distutils. I have no
intention of using distutils to build this since it ultimately must go
into a much larger external C/C++ project.
What link libraries, if any, am I missing? Is there any
documentation or examples that show how to create C callable extension
libraries, that use numpy libraries, or any examples of embedded
cython at all?
Thanks for any help in advance.
Lisandro Dalcin
This is not an issue of missing libraries... Add "-Xlinker
--export-dynamics" to your link line (in the unlikely case you do not
know what I'm talking about, look for linker option -export-dynamic
here http://linux.die.net/man/1/ld)...
--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169
SevenThunders
can't quite figure out what link libraries I need to add. By the
way, adding the link to the library libdl ( -ldl) helped with some
of the unresolved symbols. But I still am getting this problem
running the code on an Xubuntu 10 distro,
/tmp/ccX38fPA.o: In function `__Pyx_InitStrings':
/mnt/hgfs/Task21/pyembed.cpp:5201: undefined reference to
`PyUnicodeUCS4_DecodeUTF8'
I'm going to see if using distutils work for this and perhaps looking
at how it gets linked if thats possible.
On Mar 21, 7:38 pm, Lisandro Dalcin <dalc...@gmail.com> wrote:
Diez B. Roggisch
Am 22.03.2011 um 23:20 schrieb SevenThunders:
> OK that's helped a little, but I still have unresolved symbols and I
> can't quite figure out what link libraries I need to add. By the
> way, adding the link to the library libdl ( -ldl) helped with some
> of the unresolved symbols. But I still am getting this problem
> running the code on an Xubuntu 10 distro,
>
> /tmp/ccX38fPA.o: In function `__Pyx_InitStrings':
> /mnt/hgfs/Task21/pyembed.cpp:5201: undefined reference to
> `PyUnicodeUCS4_DecodeUTF8'
That loks like a totally different issue. Python can be compiled in
two flavors (well, many more, but for this discussion, it's two). They
concern the internal representation of unicode-literals. These can
either be UCS2 (2 bytes) or UCS4 (4 bytes).
It appears as if your system/libraries/whatever compile against a UCS4-
based library, but then try to *link* against one that is UCS2. So
there is something messed up there with your python. Do you by any
chance have a self-compiled Python flying around? For embedding? But
then, it seems the distutils seem to pick up the System's Python.h, or
something.
Diez
SevenThunders
is now python 2.6.5. This is in a vmware player virtual machine right
now. I'm just seeing if I can get it to work here while I don't have
access to my main machine. Interestingly the other machine also has a
local more recent python that it's using.
As for the Python.h, my compile flags should point to the local
python directory, -I~/.local/include/python2.7 . However perhaps the
other Python.h is being invoked. How do you get distutils to tell you
how it's compiling your code?
It seems like all three distros I've tried have had some weird
configuration issues that make this embedding problem a nuisance. I
haven't even gotten to the problem of forcing matplotlib to draw it's
figures when invoked from the embedded C code, but that's probably not
even a Cython issue.
Lisandro Dalcin
> Thats right. I have a local compiled Python 7 and the system python
> is now python 2.6.5. This is in a vmware player virtual machine right
> now. I'm just seeing if I can get it to work here while I don't have
> access to my main machine. Interestingly the other machine also has a
> local more recent python that it's using.
>
> As for the Python.h, my compile flags should point to the local
> python directory, -I~/.local/include/python2.7 . However perhaps the
> other Python.h is being invoked. How do you get distutils to tell you
> how it's compiling your code?
>
* Python 2.6 is my system Python
$ python2.6 -c 'from distutils import sysconfig; print
sysconfig.get_python_inc()'
/usr/include/python2.6
* Python 2.7 is a custom build installed with --prefix=/usr/local/python/2.7
$ python2.7 -c 'from distutils import sysconfig; print
sysconfig.get_python_inc()'
/usr/local/python/2.7/include/python2.7
> It seems like all three distros I've tried have had some weird
> configuration issues that make this embedding problem a nuisance.
You should take a look at Demos/embed/Makefile for a recipe about how
to query Python to get proper compiler and linker flags... This is
prepared to work with both shared lib (usually distro provided) and
static lib builds (usually custom builds) of Python.
> I
> haven't even gotten to the problem of forcing matplotlib to draw it's
> figures when invoked from the embedded C code, but that's probably not
> even a Cython issue.
>
indeed.
> On Mar 22, 6:35 pm, "Diez B. Roggisch" <de...@web.de> wrote:
>> Am 22.03.2011 um 23:20 schrieb SevenThunders:
>>
>> > OK that's helped a little, but I still have unresolved symbols and I
>> > can't quite figure out what link libraries I need to add. By the
>> > way, adding the link to the library libdl ( -ldl) helped with some
>> > of the unresolved symbols. But I still am getting this problem
>> > running the code on an Xubuntu 10 distro,
>>
>> > /tmp/ccX38fPA.o: In function `__Pyx_InitStrings':
>> > /mnt/hgfs/Task21/pyembed.cpp:5201: undefined reference to
>> > `PyUnicodeUCS4_DecodeUTF8'
>>
>> That loks like a totally different issue. Python can be compiled in
>> two flavors (well, many more, but for this discussion, it's two). They
>> concern the internal representation of unicode-literals. These can
>> either be UCS2 (2 bytes) or UCS4 (4 bytes).
>>
>> It appears as if your system/libraries/whatever compile against a UCS4-
>> based library, but then try to *link* against one that is UCS2. So
>> there is something messed up there with your python. Do you by any
>> chance have a self-compiled Python flying around? For embedding? But
>> then, it seems the distutils seem to pick up the System's Python.h, or
>> something.
>>
>> Diez
--
Alex van Houten
> I have been unable to get this to work, and I'm not seeing any
> working examples so far. I would like to embed python via cython in
> some C/C++ code. Thus I'd like to make my cython function calls from C
> ++, the opposite of what's usually done. The consumer of my code is
> not flexible in this due to the real time requirements of the C++
> code. Furthermore I have no control over the main function. I can
> not use the cython generated main() via the --embed flag
>
> After many failures due to segfaults whenever I try to call into
> cython, I figured I was not initalizing something correctly in my
> test main function, so I decided to let cython generate main for me.
What does your C++ code that calls the Cython modules look like?
I mean could you post a minimal working example?
I am still having a hard time to get this working without segfaults.
Alex.