Cython for unittesting C-Code?

[Alexander Eisenhuth]

Hello altogether,

I'm looking for a python to C wrapping technology, and wondering weather
cython is what fit the needs. I need some experiance/input/hints to
continue my exploration of Cython.

Briefly my requirements:

* Write Unittest in Python for C-Modules
* Access from python:
- enumeration literals
- static module variables
- structs, unions, typdefs
- functions
- callbacks
- stubs for calls to other C-Modules
- (As much as possible to write good whitebox unittests)

I read the documentation, and what seems clear to me, is that I need for
all access to the C-Object python code inside a pyx-File. Right? I
cannot use it in the way:

<cython_callable_object>(<cython-object>)

To get into the C-Obj code. Instead I need a python Wrapper

def python_callable(python-object):
# convert parameters and call
...

or do you see other other ways to do unittersting in cython? Do you see
limitations?

Does cython have something like automatic type conversion for structs,
enum, ...

The next question is weather cython is the best framework to "unittest
C-Code in python". I've experiences in boost.python. But compiling
boost.python extension is "not quite perfect".
What about SWIG? I've no experience in SWIG, but what makes me cringe is
the format of the *.i file.

Regards
Alexander

Here a typical code, I want to test:
---------------------- C- Header ------------------
#ifndef DEVICE_H_
#define DEVICE_H_

#include "RefTypes.h"

#define DEVICE_MAX_NUM 20

typedef ErrorStatus (*DeviceMeasureFunc)(Uint32*);

typedef struct DeviceDescriptorTag
{
Uint8 deviceNumber;
DeviceMeasureFunc deviceMeasurement;

} DeviceDescriptor;

typedef struct DeviceParamTag
{
DeviceDescriptor* deviceDescriptor;

} DeviceParam;

extern ErrorStatus Device_Init(DeviceParam*);

extern ErrorStatus Device_measure(Uint8 , Uint32*);
--------------------- C Module --------------------
#include "stdlib.h"

#include "Device.h"

static DeviceParam deviceParam;

extern ErrorStatus Device_Init(DeviceParam* param)
{
ErrorStatus retVal;

if (param != NULL &&
param->deviceDescriptor != NULL &&
param->deviceDescriptor->deviceNumber <
DEVICE_MAX_NUM &&
param->deviceDescriptor->deviceMeasurement != 0)
{
retVal = ERROR_OK;
deviceParam = *param;
}
else
{
retVal = ERROR_FAILED_INIT;

}
return retVal;
}

extern ErrorStatus Device_measure(Uint8 deviceNumber, Uint32* aiValue)
{
return deviceParam.deviceDescriptor->deviceMeasurement(aiValue);
}
----------------------------------------------------------

[Chris Barker - NOAA Federal]

On Wed, Apr 10, 2013 at 2:20 AM, Alexander Eisenhuth
<Alexander...@web.de> wrote:

> I'm looking for a python to C wrapping technology, and wondering weather
> cython is what fit the needs. I need some experiance/input/hints to continue
> my exploration of Cython.
>
> Briefly my requirements:
>
> * Write Unittest in Python for C-Modules
> * Access from python:
> - enumeration literals
> - static module variables
> - structs, unions, typdefs
> - functions
> - callbacks
> - stubs for calls to other C-Modules
> - (As much as possible to write good whitebox unittests)

Cython can do all of this, but _may_ not be the best choice....

> I read the documentation, and what seems clear to me, is that I need for all
> access to the C-Object python code inside a pyx-File. Right? I cannot use it
> in the way:
>
> <cython_callable_object>(<cython-object>)

I"m not sure I follow what this is intended to mean

> To get into the C-Obj code. Instead I need a python Wrapper
>
> def python_callable(python-object):
> # convert parameters and call

yes, though the "convert parameters" is simple for the simple cases.

> Does cython have something like automatic type conversion for structs, enum,

not out of the box -- though there have been some efforts along this line. See:

http://wiki.cython.org/AutoPxd

for an overview of options. None of them are all that mature, but I
think someone reported success with cwrap on this list recently.

And there is this new project that looks pretty promising:

https://s3.amazonaws.com/xdress/index.html

> The next question is weather cython is the best framework to "unittest
> C-Code in python".

if you only goal is testing the C code, then I'm not sre it's the best
way to go. I"d take a good look at ctypes for that (and I'm pretty
sure there is a tool to auto-generate ctypes wrappers for C code)
However, if you want to be able to call your C code from python for
production purposes, then it's a great choice.

The big difference is that ctypes allows (and requires) you to call C
more or less directly. The strength is in is that you can write only
in Pyton, and you can call dynamic libraries without compiling
anything or having access to the origina code -- great for calling the
occasional system lib. But a nice python-C wrapper with full features
always requires to "wrapper code" - code to provide amore python ic
interface, handle the conversion of complex classes and types, etc.
with ctypes, you write that code in Python, which can be fine, but
you're never going to get full performance that way. With Cython, you
write that code in Cython, and can get full c-like performance.

This may not be any advantage to you for testing only.

> I've experiences in boost.python. But compiling
> boost.python extension is "not quite perfect".

I think Boost.pyton is total overkill for testing.

> What about SWIG? I've no experience in SWIG, but what makes me cringe is the
> format of the *.i file.

yes, a bit painfull, though it doe smake the easy stuff easy, so if
most of your code is easy, then it could be a fine option -- I"d take
a look at SIP as an alternative as well.

My thoughts: if you are uysing this primarily (or only) for testing,
you don't need the best performance, or nicest Pyton API -- but you
probably do want auto-generation of the wrappers. So I"d see if any of
the cython-generating tools work for you, and if not, try ctypes of
SIP or SWIG.

(my survey of the cython-generating tools indicates that C support may
be pretty good -- I personaly need to wrap C++, and so haven't tried
them, their C++ support is weak (except maybe xdress, which is brand
new and I haven't tried yet)

HTH,
-Chris



--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Daniele Nicolodi]

On 10/04/2013 17:37, Chris Barker - NOAA Federal wrote:
> if you only goal is testing the C code, then I'm not sre it's the best
> way to go. I"d take a good look at ctypes for that (and I'm pretty
> sure there is a tool to auto-generate ctypes wrappers for C code)
> However, if you want to be able to call your C code from python for
> production purposes, then it's a great choice.

I would try the newcomer cffi too: https://cffi.readthedocs.org/

It call C functions without requiring a compilation stage, with a
mechanism similar to ctypes, but it is capable of parsing C headers for
function declarations, alleviating the burden to have to define
parameters types at the wrapper level.

PS: I'm wondering if there may be some interaction between the cffi
parser and cython for automagic generation of wrapper code...

Cheers,
Daniele

[Chris Barker - NOAA Federal]

On Wed, Apr 10, 2013 at 8:51 AM, Daniele Nicolodi <dan...@grinta.net> wrote:
> PS: I'm wondering if there may be some interaction between the cffi
> parser and cython for automagic generation of wrapper code...

cffi uses: pycparser >= 2.06: http://code.google.com/p/pycparser/

which does look really promising -- no C++ support, but if you don't need that.

I really like no depending on complex external on-python tools like
gcc or doxygen, or clang to do the parsing.

[Stefan Behnel]

Chris Barker - NOAA Federal, 10.04.2013 19:43:

> On Wed, Apr 10, 2013 at 8:51 AM, Daniele Nicolodi wrote:
>> PS: I'm wondering if there may be some interaction between the cffi
>> parser and cython for automagic generation of wrapper code...
>
> cffi uses: pycparser >= 2.06: http://code.google.com/p/pycparser/
>
> which does look really promising -- no C++ support, but if you don't need that.
>
> I really like no depending on complex external on-python tools like
> gcc or doxygen, or clang to do the parsing.

Note that cffi depends on a C compiler being available at runtime in some
cases.

Stefan
u

[francis]

> I'm looking for a python to C wrapping technology,

Just mention some:

wraptypes from pyglet
->
https://code.google.com/p/pyglet/source/browse/tools/wraptypes/?name=image_refactor

python cffi
-> https://cffi.readthedocs.org/en/release-0.6/

ctypesgen
-> https://code.google.com/p/ctypesgen/


Regards,
francis

[Alexander Eisenhuth]

Thanks a lot for your fast responses. As I need in any case a compiler I
stay at cython and have a look at the automatic wrapper generators for
cython you mentioned.

For any further hints I thank in advance ...

Alexander

[Anthony Scopatz]

On Wednesday, April 10, 2013 10:37:43 AM UTC-5, Chris Barker - NOAA Federal wrote:

[snip]

 

(my survey of the cython-generating tools indicates that C support may
be pretty good -- I personaly need to wrap C++, and so haven't tried
them, their C++ support is weak (except maybe xdress, which is brand
new and I haven't tried yet)

Hey Chris, 

Just to jump in here, automatically wrapping arbitrary  C++ classes was the original point of xdress and I have expanded from there based on need (functions, dtypes, views into STL containers .  If other people have other needs and use cases that aren't currently supported I am very willing to continue to develop xdress in these directions.  Of course PRs are always welcome too ;).

Be Well

Anthony

[Chris Barker - NOAA Federal]

On Mon, Apr 15, 2013 at 8:44 AM, Anthony Scopatz <sco...@gmail.com> wrote:

> Just to jump in here, automatically wrapping arbitrary C++ classes was the
> original point of xdress and I have expanded from there based on need
> (functions, dtypes, views into STL containers . If other people have other
> needs and use cases that aren't currently supported I am very willing to
> continue to develop xdress in these directions. Of course PRs are always
> welcome too ;).

Great -- looks very promising.

By project-at-hand is in mid-stream with hand-written Cython wrappers,
so we'll see. But I expect that we'll give xdress a try at some point.