Wrap C/C++ lib with Cython

[Yuqing Zhang]

Hi,

 

I’m new to Cython and I’m trying to use Cython to wrap a C/C++ lib, so I can use the lib in my Python project.

 

I made a small example. There is a lib file called LibTest.lib.  The head file is like:

 

//test.h

#ifndef TEST_H

#define TEST_H

 

int add(int a, int b);

int multipy(int a, int b);

 

#endif

 

My .pxd file is like:

 

#ctest.pxd

cdef extern from "test.h":

      int add(int,int)

      int multipy(int,int)

 

And the .pyx file:

 

#ctest.pyx

cimport ctest

cimport python_exc

 

print ctest.add(2,3)

 

The setup.py is:

 

from distutils.core import setup

from distutils.extension import Extension

from Cython.Distutils import build_ext

 

ext_modules = [Extension("ctest",

                         ["ctest.pyx"],

                         include_dirs=[r’..\RefLibTest'],

                         library_dirs=[r’..\release'],

                         libraries=['LibTest']

                         )]

 

setup(

  name = 'ctest',

  cmdclass = {'build_ext': build_ext},

  ext_modules = ext_modules

)

 

So now I got a ctest.pyd and I made a python call:

Import ctest

I got the error message:

import ctest

File "ctest.pyx", line 4, in init ctest (ctest.c:468)

print ctest.add(2,3)

NameError: ctest

So I added “language=’c++’” in the ext_modules. But this time it failed to build, and I got the error message:

g++: build\temp.win32-2.6\Release\libctest.a: No such file or directory

error: command 'g++' failed with exit status 1

Can anyone help me?

Thank you!

 

Yuqing Zhang

AnyBody Technology A/S,  www.anybodytech.com
Niels Jernes Vej 10, 9220, Aalborg Øst, Denmark

Office: +45 96 35 42 90

Mobile: +45 60 71 07 65

Email: y...@anybodytech.com

 

[Robert Bradshaw]

Don't cimport ctest in ctest.pyx. Instead, you should just do

#ctest.pyx
cimport python_exc

print add(2,3)

Either that, or name the .pxd and .pyx file something different.

- Robert

[Yuqing Zhang]

Well, now I merged .pxd and .pyx file.

Test.pyx:

cdef extern from "test.h":
int add(int a,int b)
int multipy(int a,int b)

print add(2,3)

Then I got a linking error when I tried to build it.

Undefined reference to 'add'

A little bit confused...

 
Yuqing Zhang
AnyBody Technology A/S,  www.anybodytech.com
Niels Jernes Vej 10, 9220, Aalborg Øst, Denmark
Office: +45 96 35 42 90
Mobile: +45 60 71 07 65
Email: y...@anybodytech.com

[Dag Sverre Seljebotn]

Yuqing Zhang wrote:
> Well, now I merged .pxd and .pyx file.
>
> Test.pyx:
>
> cdef extern from "test.h":
> int add(int a,int b)
> int multipy(int a,int b)
>
> print add(2,3)
>
> Then I got a linking error when I tried to build it.
>
> Undefined reference to 'add'
>
> A little bit confused...
>

This is a problem with your build setup, such as "add" not being found
in the "LibTest" library or similar. To have any hope of getting any
help you will need to post the full output of your build, what compiler
you are using, etc. etc.

Dag Sverre

[Yuqing Zhang]

I tried something today.

Test.h:
------------------------------------------------------------
#ifndef TEST_H
#define TEST_H

int add(int a, int b);
int multipy(int a, int b);

#endif
--------------------------------------------------------------

Test.cpp:
--------------------------------------------------------------
#include "test.h"

int add(int a, int b)

{
return a+b;
}

int multipy(int a, int b)

{
return a*b;
}
--------------------------------------------------------------

Then I used g++ to compile and build it.
-------------------------------------------------------------
g++ -c test.cpp -o libtest.o
ar rcs libtest.a libtest.o
--------------------------------------------------------------

So now I got a static library called libtest.a.

Test.pyx:
----------------------------------------------------------------

cdef extern from "test.h":
int add(int a,int b)
int multipy(int a,int b)

print add(2,3)
-----------------------------------------------------------------

Setup.py:
------------------------------------------------------------------

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [Extension("test",
["test.pyx"],
language='c++',
include_dirs=[r'.'],
library_dirs=[r'.'],
libraries=['libtest']
)]

setup(
name = 'test',

cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)

------------------------------------------------------------------

The I called:
python setup.py build_ext --compiler=mingw32 --inplace

The output was:
-------------------------------------------------------------------------------------------------------------------
running build_ext
cythoning test.pyx to test.cpp
building 'test' extension
creating build
creating build\temp.win32-2.6
creating build\temp.win32-2.6\Release
C:\Program Files\pythonxy\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -I. -IC:\
Python26\include -IC:\Python26\PC -c test.cpp -o build\temp.win32-2.6\Release\te
st.o
writing build\temp.win32-2.6\Release\test.def
C:\Program Files\pythonxy\mingw\bin\g++.exe -mno-cygwin -mdll -static --entry _D
llMain@12 --output-lib build\temp.win32-2.6\Release\libtest.a --def build\temp.w
in32-2.6\Release\test.def -s build\temp.win32-2.6\Release\test.o -L. -LC:\Python
26\libs -LC:\Python26\PCbuild -ltest -lpython26 -lmsvcr90 -o test.pyd
g++: build\temp.win32-2.6\Release\libtest.a: No such file or directory

error: command 'g++' failed with exit status 1

---------------------------------------------------------------------------------------------------------------------

I also tried to use libraries=['test'] instead of libraries=['libtest']. It gave me the same errors.

Any clue about those?

 
Yuqing Zhang
AnyBody Technology A/S,  www.anybodytech.com
Niels Jernes Vej 10, 9220, Aalborg Øst, Denmark
Office: +45 96 35 42 90
Mobile: +45 60 71 07 65
Email: y...@anybodytech.com

-----Original Message-----
From: cython...@googlegroups.com [mailto:cython...@googlegroups.com] On Behalf Of Dag Sverre Seljebotn
Sent: Wednesday, January 20, 2010 10:32 AM
To: cython...@googlegroups.com
Subject: Re: [cython-users] Wrap C/C++ lib with Cython

[Dag Sverre Seljebotn]

Well, first thing I'd try is to get rid of "." from library_path -- try
to use os.getcwd() instead.

Dag Sverre

>
>
> Yuqing Zhang
> AnyBody Technology A/S, www.anybodytech.com

> Niels Jernes Vej 10, 9220, Aalborg �st, Denmark

> Office: +45 96 35 42 90
> Mobile: +45 60 71 07 65
> Email: y...@anybodytech.com
>
>
> -----Original Message-----
> From: cython...@googlegroups.com [mailto:cython...@googlegroups.com] On Behalf Of Dag Sverre Seljebotn
> Sent: Wednesday, January 20, 2010 10:32 AM
> To: cython...@googlegroups.com
> Subject: Re: [cython-users] Wrap C/C++ lib with Cython
>
> Yuqing Zhang wrote:
>> Well, now I merged .pxd and .pyx file.
>>
>> Test.pyx:
>>
>> cdef extern from "test.h":
>> int add(int a,int b)
>> int multipy(int a,int b)
>>
>> print add(2,3)
>>
>> Then I got a linking error when I tried to build it.
>>
>> Undefined reference to 'add'
>>
>> A little bit confused...
>>
> This is a problem with your build setup, such as "add" not being found
> in the "LibTest" library or similar. To have any hope of getting any
> help you will need to post the full output of your build, what compiler
> you are using, etc. etc.
>
> Dag Sverre

--
Dag Sverre

[Yuqing Zhang]

Thanks, but os.getcwd() doesn't work.

It is quite weird to me that it complained there is no libtest.a in 'build\temp.win32-2.6\Release'. Isn't that the build folder generated by Cython itself? Why did Cython try to look for the libtest.a there?

I checked the Cython tutorials, in the chapter 'Using C libraries', it says:

we can extend the script to include the necessary setup for building against the external C library. Assuming it's installed in the normal places (e.g. under /usr/lib and /usr/include on a Unix-like system), we could simply change the extension setup

If it is not installed in a 'normal' location, users can provide the required parameters externally by passing appropriate C compiler flags, such as:
CFLAGS="-I/usr/local/otherdir/calg/include" \
LDFLAGS="-L/usr/local/otherdir/calg/lib" \
python setup.py build_ext -i

How to set CFLAGS, LDFLAGS? Should I make a .mak file?

 
Yuqing Zhang
AnyBody Technology A/S,  www.anybodytech.com

Niels Jernes Vej 10, 9220, Aalborg Øst, Denmark

Dag Sverre

> Niels Jernes Vej 10, 9220, Aalborg Øst, Denmark

[Yuqing Zhang]

I tried to download the source codes in 'Wrapping C++ classes in Cython' and I got the same error.

python setup.py build_ext --compile=mingw32 --inplace

running build_ext
cythoning rectangle.pyx to rectangle.cpp
warning: C:\yz\Projects\RLM\ALicensesCython\src\Rect\rectangle.pyx:41:6: String
literals will no longer be Py3 bytes in Cython 0.12.
building 'rectangle' extension

creating build
creating build\temp.win32-2.6
creating build\temp.win32-2.6\Release

C:\Program Files\pythonxy\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Pyt
on26\include -IC:\Python26\PC -c rectangle.cpp -o build\temp.win32-2.6\Release\
ectangle.o
rectangle.cpp: In function 'void __Pyx_RaiseArgtupleInvalid(const char*, int, P
_ssize_t, Py_ssize_t, Py_ssize_t)':
rectangle.cpp:1486: warning: unknown conversion type character 'z' in format
rectangle.cpp:1486: warning: format '%s' expects type 'char*', but argument 5 h
s type 'Py_ssize_t'
rectangle.cpp:1486: warning: unknown conversion type character 'z' in format
rectangle.cpp:1486: warning: too many arguments for format
C:\Program Files\pythonxy\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Pyt
on26\include -IC:\Python26\PC -c cpp_rect.cpp -o build\temp.win32-2.6\Release\c
p_rect.o
writing build\temp.win32-2.6\Release\rectangle.def

C:\Program Files\pythonxy\mingw\bin\g++.exe -mno-cygwin -mdll -static --entry _

llMain@12 --output-lib build\temp.win32-2.6\Release\librectangle.a --def build\
emp.win32-2.6\Release\rectangle.def -s build\temp.win32-2.6\Release\rectangle.o
build\temp.win32-2.6\Release\cpp_rect.o -LC:\Python26\libs -LC:\Python26\PCbuil
-lpython26 -lmsvcr90 -o rectangle.pyd
g++: build\temp.win32-2.6\Release\librectangle.a: No such file or directory

error: command 'g++' failed with exit status 1

[Robert Bradshaw]

On Jan 21, 2010, at 1:28 AM, Yuqing Zhang wrote:

> Thanks, but os.getcwd() doesn't work.
>
> It is quite weird to me that it complained there is no libtest.a in
> 'build\temp.win32-2.6\Release'. Isn't that the build folder
> generated by Cython itself? Why did Cython try to look for the
> libtest.a there?

This is more of a distutils on cygwin question... Cython turns a .pyx
file into a .c file, then the distutils stuff in setup.py does the rest.

> I checked the Cython tutorials, in the chapter 'Using C libraries',
> it says:
>
> we can extend the script to include the necessary setup for building
> against the external C library. Assuming it's installed in the
> normal places (e.g. under /usr/lib and /usr/include on a Unix-like
> system), we could simply change the extension setup
>
> If it is not installed in a 'normal' location, users can provide the
> required parameters externally by passing appropriate C compiler
> flags, such as:
> CFLAGS="-I/usr/local/otherdir/calg/include" \
> LDFLAGS="-L/usr/local/otherdir/calg/lib" \
> python setup.py build_ext -i
>
> How to set CFLAGS, LDFLAGS? Should I make a .mak file?

Do you mean a Makefile? You can do that, but it's usually more
complicated.

Can you try calling either your external library or your .pyx file
something other than "test"--it may be that it's mixing up the names
when trying to create two libraries with the same name, and at the
very least would make the errors easier to diagnose.

I created http://hg.cython.org/cython-devel/file/tip/Demos/libraries ,
see if that works for you.

- Robert

[josef...@gmail.com]

It works without problems for me on WindowsXP with MingW and no cygwin.

Thanks,
Josef

>
> - Robert
>
>

[Yuqing Zhang]

Thanks Robert.

The demo works. Now I have no problem to link a C static library. However, I still have problem to link a C++ library. I simply renamed the mymath.c to mymath.cpp, and changed line 17 in setup.py to

assert os.system("g++ -c mymath.cpp -o mymath.o") == 0

and added language="C++" in ext_modules. Then I got the following error:

building libmymath.a
running build_ext
cythoning call_mymath.pyx to call_mymath.cpp
building 'call_mymath' extension

creating build
creating build\temp.win32-2.6
creating build\temp.win32-2.6\Release

C:\Program Files (x86)\pythonxy\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC
:\Users\yz\Projects\AnyBody\RLM\Demos\PythonWrapper\ALicensesCython\src\math -IC
:\Python26\include -IC:\Python26\PC -c call_mymath.cpp -o build\temp.win32-2.6\R
elease\call_mymath.o
writing build\temp.win32-2.6\Release\call_mymath.def
C:\Program Files (x86)\pythonxy\mingw\bin\dllwrap.exe -mno-cygwin -mdll -static
--entry _DllMain@12 --output-lib build\temp.win32-2.6\Release\libcall_mymath.a -
-def build\temp.win32-2.6\Release\call_mymath.def -s build\temp.win32-2.6\Releas
e\call_mymath.o -LC:\Users\yz\Projects\AnyBody\RLM\Demos\PythonWrapper\ALicenses
Cython\src\math -LC:\Python26\libs -LC:\Python26\PCbuild -lmymath -lpython26 -lm
svcr90 -o call_mymath.pyd
build\temp.win32-2.6\Release\call_mymath.o:call_mymath.cpp:(.eh_frame+0x11): und
efined reference to `__gxx_personality_v0'
collect2: ld returned 1 exit status
dllwrap: gcc exited with status 1
error: command 'dllwrap' failed with exit status 1

The error is quite obvious, Cython tried to use gcc to link the C++ lib. How can I make Cython to use g++ to link a C++ lib?

-----Original Message-----
From: cython...@googlegroups.com [mailto:cython...@googlegroups.com] On Behalf Of Robert Bradshaw
Sent: 22. januar 2010 07:36
To: cython...@googlegroups.com
Subject: Re: [cython-users] Wrap C/C++ lib with Cython

[Robert Bradshaw]

On Jan 25, 2010, at 2:11 AM, Yuqing Zhang wrote:

> Thanks Robert.
>
> The demo works. Now I have no problem to link a C static library.
> However, I still have problem to link a C++ library. I simply
> renamed the mymath.c to mymath.cpp, and changed line 17 in setup.py to
>
> assert os.system("g++ -c mymath.cpp -o mymath.o") == 0
>
> and added language="C++" in ext_modules. Then I got the following
> error:
>
> building libmymath.a
> running build_ext
> cythoning call_mymath.pyx to call_mymath.cpp

Cython's job is done at this point, the rest is a distutils question.

I don't know. One other thing to try would be to do
extra_objects=["libmymath.a"] rather than listing it in the libraries
parameter, but short of that I think you'll have to ask on the
distutils list. Maybe C++ is stricter about trying to link static vs.
shared libraries?

- Robert

[Samuel Bronson]

On Mon, Jan 25, 2010 at 1:20 PM, Robert Bradshaw
<robe...@math.washington.edu> wrote:
> On Jan 25, 2010, at 2:11 AM, Yuqing Zhang wrote:

>> build\temp.win32-2.6\Release\call_mymath.o:call_mymath.cpp:(.eh_frame+0x11):
>> und
>> efined reference to `__gxx_personality_v0'
>> collect2: ld returned 1 exit status
>> dllwrap: gcc exited with status 1
>> error: command 'dllwrap' failed with exit status 1

> I don't know. One other thing to try would be to do

> extra_objects=["libmymath.a"] rather than listing it in the libraries
> parameter, but short of that I think you'll have to ask on the distutils
> list. Maybe C++ is stricter about trying to link static vs. shared
> libraries?

Actually, I believe the problem is that the C++ runtime library needs
to be pulled in (be it statically or dynamically), which happens by
default with g++, but not with gcc.

Perhaps adding the distutils equivalent of -lstdc++ would solve the problem?

[Robert Bradshaw]

Great insight, that would be libraries=['stdc++', ...].

- Robert