Wrapping C libraries for Python - Does cffi work with cython?

[Sarvi Shanmugham]

Some one on this alias suggested cffi as a better option to create python wrappers to C libraries.

I have been trying it and it is awesome. So far the best approach, by far, to generating a python wrapper around a C shared library.

But I still like Cython performance for optimizing/compiling much of my python code where needed.

Hence the question, Does Cython play well with CFFI??

Can Cython compile the CFFI and CFFI wrapper code, and Can Compiled Cython code invoke the CFFI wrapper directly??

Sarvi

[Stefan Behnel]

Sarvi Shanmugham, 16.09.2012 02:16:

> Some one on this alias suggested cffi as a better option to create python
> wrappers to C libraries.
>
> I have been trying it and it is awesome. So far the best approach, by far,
> to generating a python wrapper around a C shared library.

Well, it's not really all that different from Cython code, just slower (in
CPython) and without support for C++. All that it really gives you is
in-place declarations for external C APIs and C function signatures that
you can copy over from C header files directly. In Cython, you'll have to
adapt some of them to Cython syntax first.

> But I still like Cython performance for optimizing/compiling much of my
> python code where needed.
>
> Hence the question, Does Cython play well with CFFI??

Yes, in the sense that both don't interfere.

> Can Cython compile the CFFI and CFFI wrapper code, and Can Compiled Cython
> code invoke the CFFI wrapper directly??

Short answer: if you implement that, then yes. There isn't a technical
problem, it just has to be done. Look at the discussions on the core
developer mailing list regarding fast function dispatch and also CEP 1000.

http://wiki.cython.org/enhancements/cep1000

You may want to ask on the cython-dev list regarding its current status.

Stefan

[Dag Sverre Seljebotn]

On 09/16/2012 08:00 AM, Stefan Behnel wrote:
> Sarvi Shanmugham, 16.09.2012 02:16:
>> Some one on this alias suggested cffi as a better option to create python
>> wrappers to C libraries.
>>
>> I have been trying it and it is awesome. So far the best approach, by far,
>> to generating a python wrapper around a C shared library.
>
> Well, it's not really all that different from Cython code, just slower (in
> CPython) and without support for C++. All that it really gives you is
> in-place declarations for external C APIs and C function signatures that
> you can copy over from C header files directly. In Cython, you'll have to
> adapt some of them to Cython syntax first.

There's a critical piece that's missing from Cython: The ability to
automatically export declarations to Python space. Something like a
"cpdef extern:" block. (Unless that was added recently.)

This means there's roughly 2x as much code as cffi.

Also since pointers etc. aren't converted automatically to Python space
by Cython (I assume you can get "pointer objects" in the cffi runtime
library) you need to spend the time to map these to Pythonic constructs
in Cython code rather than Python code.

Dag

[Sarvi Shanmugham]

On Saturday, September 15, 2012 11:00:13 PM UTC-7, Stefan Behnel wrote:

Sarvi Shanmugham, 16.09.2012 02:16:
> Some one on this alias suggested cffi as a better option to create python
> wrappers to C libraries.
>
> I have been trying it and it is awesome. So far the best approach, by far,
> to generating a python wrapper around a C shared library.

Well, it's not really all that different from Cython code, just slower (in
CPython) and without support for C++. All that it really gives you is
in-place declarations for external C APIs and C function signatures that
you can copy over from C header files directly. In Cython, you'll have to
adapt some of them to Cython syntax first.

That I think is an over simplification. I initially started with using Cython to write my Python wrappers for a set of libraries I had.

And trust after many weeks of effort and mail exchanges with Cython dev I switched to CFFI.

Infact CFFI was suggested on the Cython alias. 

A lot of my trouble was exposing C structures for creation/reference and use from Python. This was simple and needed a lot of cython code.

And I can tell you after using, CFFI was a breeeze. all I had to do was cut and paste the entire C header file as is, then replace a few #defines values with "..."

And I was done. I now have an Python API that is a very easy to use and with all the flexibility I need.

Don't get me wrong. 

I am still a cython lover.

Except not for its C shared library wrapping skills, but for the fact that I can compile portions of my code and optimize for performance.

 

My question hence, is if I were to write a piece of Cython code that called shared library wrapped through CFFI, would it cost me performance?

> But I still like Cython performance for optimizing/compiling much of my
> python code where needed.
>
> Hence the question, Does Cython play well with CFFI??

Yes, in the sense that both don't interfere.


> Can Cython compile the CFFI and CFFI wrapper code, and Can Compiled Cython
> code invoke the CFFI wrapper directly??

Short answer: if you implement that, then yes. There isn't a technical
problem, it just has to be done. Look at the discussions on the core
developer mailing list regarding fast function dispatch and also CEP 1000.

http://wiki.cython.org/enhancements/cep1000

You may want to ask on the cython-dev list regarding its current status.

Not sure I understand the answer, but will research cep1000 out.

The question I had in mind was, If I had say a shared math library libmath.so and wrapped it with a CFFI based python wrapper.

And then wrote a piece of cython code that invoked that CFFI based python wrapper, how will my performance compared to 

writing the python wrapper for libmath.so with  cython cdef or cpdef wrapper function definitions.

Thanks,

Sarvi

Stefan

[Chris Barker]

On Sat, Sep 15, 2012 at 11:00 PM, Stefan Behnel <stef...@behnel.de> wrote:

> Well, it's not really all that different from Cython code, just slower (in
> CPython) and without support for C++. All that it really gives you is
> in-place declarations for external C APIs and C function signatures that
> you can copy over from C header files directly. In Cython, you'll have to
> adapt some of them to Cython syntax first.

Which is actually a pretty big deal -- or a big pain in the @%$ anyway.

I've seen reference on this list of work for auto-generating wrappers,
anyone know the status of those projects?

While a robust feature-complete wrapper-generator for C++ is a massive
and challenging undertaking, it seems that auto-generating the easy
stuff -- stucts, simple functions, should be pretty straightforward,
and still a pretty good labor-saver.

maybe CFFI code could be borrowed?

-Chris


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Robert Bradshaw]

On Mon, Sep 17, 2012 at 11:43 AM, Chris Barker <chris....@noaa.gov> wrote:
> On Sat, Sep 15, 2012 at 11:00 PM, Stefan Behnel <stef...@behnel.de> wrote:
>
>> Well, it's not really all that different from Cython code, just slower (in
>> CPython) and without support for C++. All that it really gives you is
>> in-place declarations for external C APIs and C function signatures that
>> you can copy over from C header files directly. In Cython, you'll have to
>> adapt some of them to Cython syntax first.
>
> Which is actually a pretty big deal -- or a big pain in the @%$ anyway.
>
> I've seen reference on this list of work for auto-generating wrappers,
> anyone know the status of those projects?
>
> While a robust feature-complete wrapper-generator for C++ is a massive
> and challenging undertaking, it seems that auto-generating the easy
> stuff -- stucts, simple functions, should be pretty straightforward,
> and still a pretty good labor-saver.

Yes, for sure. There was a GSoC project by Philip Heron this summer to
do just that. The code is up at
https://github.com/robertwb/cython/tree/pxd-gcc-generation , and it
does work, provided you can get gcc-python-plugin to work.

Really, the hard part here is parsing the C (and especially C++), and
the goal of this project was to use gcc itself to do this.
Unfortunately the python plugin (and gcc plugin API) still seems to be
under development and hard to build.

> maybe CFFI code could be borrowed?

Looks like cffi uses http://code.google.com/p/pycparser/ . I really
don't think it'd be that hard to write something that translates
(most) C headers into pxd headers, and C++ provided a suitable parsing
library. I haven't had time to do it myself, so I guess I shouldn't be
surprised no one else has either... One thing that's unclear is how to
best handle the parsing (as gcc plugins didn't work as well as we had
hoped, though I do think things will improve with time) but writing a
.h -> .pxd translator is only loosely coupled with Cython itself, so
anyone could just write such a tool and then we could look at how best
to integrate it once it becomes clear what the best approach/set of
dependencies is.

- Robert

[Chris Barker]

On Mon, Sep 17, 2012 at 12:51 PM, Robert Bradshaw <robe...@gmail.com> wrote:

>> I've seen reference on this list of work for auto-generating wrappers,
>> anyone know the status of those projects?

> Yes, for sure. There was a GSoC project by Philip Heron this summer to
> do just that. The code is up at
> https://github.com/robertwb/cython/tree/pxd-gcc-generation , and it
> does work, provided you can get gcc-python-plugin to work.

> Really, the hard part here is parsing the C (and especially C++), and
> the goal of this project was to use gcc itself to do this.
> Unfortunately the python plugin (and gcc plugin API) still seems to be
> under development and hard to build.

and not helpful to folks using MS compilers, thought he code
generation is an independent step anyway.

> Looks like cffi uses http://code.google.com/p/pycparser/ . I really
> don't think it'd be that hard to write something that translates
> (most) C headers into pxd headers, and C++ provided a suitable parsing
> library.

I don't follow -- typo maybe? did you mean g++ ?

there is also Doxygen and SWIG. I haven't looked at Doxegen's XML, but
maybe there is promise there. I don't think I want to go down the SWIG
path, that's exactly what I'm trying to avoid by doing this! (anone
looked at SIP???)

> but writing a
> .h -> .pxd translator is only loosely coupled with Cython itself, so
> anyone could just write such a tool and then we could look at how best
> to integrate it once it becomes clear what the best approach/set of
> dependencies is.

yup -- I was hoping someone had a good start -- maybe I'll see what it
takes to get gcc-python-plugin to work.

[David Cournapeau]

As the risk of sounding like a broken record, I strongly suggest
people with time to work on this to use clang and its python cindex
API for this. pycparser has the advantage of being in pure python, but
I suspect will never deal with a lot of corner cases as well as clang
does.

David

[Chris Barker]

On Mon, Sep 17, 2012 at 2:41 PM, David Cournapeau <cour...@gmail.com> wrote:
> As the risk of sounding like a broken record, I strongly suggest
> people with time to work on this to use clang and its python cindex
> API for this. pycparser has the advantage of being in pure python, but
> I suspect will never deal with a lot of corner cases as well as clang
> does.

nor do they intend to support C++ -- in fact, they recommend clang:

http://eli.thegreenplace.net/2011/07/03/parsing-c-in-python-with-clang/

David,

Didn't you make a start at a wrapper-generator? What's it's status?

[Alex Leach - Gmail]

On Monday 17 Sep 2012 12:51:48 Robert Bradshaw wrote:

 

> Looks like cffi uses http://code.google.com/p/pycparser/ . I really

> don't think it'd be that hard to write something that translates

> (most) C headers into pxd headers, and C++ provided a suitable parsing

> library. I haven't had time to do it myself, so I guess I shouldn't be

> surprised no one else has either...

 

There is Enthought's cwrap utility: https://github.com/enthought/cwrap/

 

It looks a little neglected, esp. wrt. docs, but I'm sure it was made just for this. The setup.py's description:-

"Automatical generate Cython wrappers from C header files"

 

KR,

Alex

[Robert Bradshaw]

http://wiki.cython.org/AutoPxd

[Chris Barker]

On Wed, Sep 19, 2012 at 12:59 PM, Robert Bradshaw <robe...@gmail.com> wrote:

> http://wiki.cython.org/AutoPxd


Thanks Robert -- very helpful putting all that in one place.

Now I need to figure how how much of a pain gcc-xml is to get running...

[David Cournapeau]

On Mon, Sep 17, 2012 at 11:04 PM, Chris Barker <chris....@noaa.gov> wrote:
> On Mon, Sep 17, 2012 at 2:41 PM, David Cournapeau <cour...@gmail.com> wrote:
>> As the risk of sounding like a broken record, I strongly suggest
>> people with time to work on this to use clang and its python cindex
>> API for this. pycparser has the advantage of being in pure python, but
>> I suspect will never deal with a lot of corner cases as well as clang
>> does.
>
> nor do they intend to support C++ -- in fact, they recommend clang:
>
> http://eli.thegreenplace.net/2011/07/03/parsing-c-in-python-with-clang/
>
> David,
>
> Didn't you make a start at a wrapper-generator? What's it's status?

Last time I looked at it (clang 2.9 or 3.0, can't remember), the
cindex API was missing some crucial parts (to resolve enum, for
example), and I needed to extend it to get that working. I never took
the time to continue since then. I still think someone could do at
least as much as my cython-codegen using clang in less than a WE.

It would be nice to then implement this as a frontend for cwrap
instead of some ad-hoc hack ala cython-codegen.

David

[Gour]

On Sun, 16 Sep 2012 23:40:31 -0700 (PDT)
Sarvi Shanmugham <sarv...@gmail.com> wrote:

Hello Sarvi,

it's a bit late inquiry, but didn't find it earlier...

> That I think is an over simplification. I initially started with using
> Cython to write my Python wrappers for a set of libraries I had. And trust
> after many weeks of effort and mail exchanges with Cython dev I switched to
> CFFI.

I wonder how flexible is CFFI in providing more higher-level pythonic API to C
code and not just a wrapper?

> And I can tell you after using, CFFI was a breeeze. all I had to do was cut
> and paste the entire C header file as is, then replace a few #defines values
> with "..." And I was done. I now have an Python API that is a very easy to
> use and with all the flexibility I need.

When you say 'Python API', do you mean not just wrapper over C API, but
higher-level one?


Sincerely,
Gour

--
Even if you are considered to be the most sinful of all sinners,
when you are situated in the boat of transcendental knowledge
you will be able to cross over the ocean of miseries.

http://www.atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

[Sarvi Shanmugham]

The Python wrapper API feels quite pythonic. But it does have some see elements to it, because you are 

working to invoke the C-API underneath and that part is innevitable.

When working with the direct Python API, you do have to remember you working a C wrapped API, specially when 

working with pointers and arrays and char* etc. But the way C-interactions have been wrapped is quite reasonably pythonic.

I'd say its a good tradeoff between speed of wrapping and convenience of the API.

Sarvi