Cython vs Ctypes - For wrapping C++ code

[vybhav ramachandran]

Hi,

I'm relatively new to the world of wrapping C++ code to be used in Python, but I've come across 2 methods to do this:

* Use Cython

* Use Ctypes

From the outside, if it's just wrapping that I need, Ctypes looks to be a more convenient option as the developer doesn't have to spend a lot of time wrapping each C++ function manually (like they would have to do in cython).

But, it seems too good to be true and I'm looking for some downsides to this approach and why Cython could be better.

Can anyone please share their experience and some non-obvious differences between the two?

Thanks,

Vybhv

[da-woods]

For Ctypes you're going to need to be guessing the name mangling. That means it's unlikely to work across different C++ compilers and may well stop working as compilers are improved or new C++ standards emerge. Alternatively you can wrap everything with 'extern "C"', but then you've limited the features you can use.

As you note Cython requires you spend time wrapping everything.

Pybind11 is also a good option (but you'll also need to spend time wrapping everything).

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/854e7782-5465-4fe3-a3cb-0bfd90242aecn%40googlegroups.com.

[Stefan Behnel]

vybhav ramachandran schrieb am 12.11.20 um 03:08:

> I'm relatively new to the world of wrapping C++ code to be used in Python,
> but I've come across 2 methods to do this:
>
> * Use Cython
> * Use Ctypes
>
> From the outside, if it's just wrapping that I need, Ctypes looks to be a
> more convenient option as the developer doesn't have to spend a lot of time
> wrapping each C++ function manually (like they would have to do in cython).

David Woods already mentioned the difference between connecting to the ABI
(which ctypes uses) and the API (which Cython uses), especially since the
ABI is not defined in a portable way for C++ libraries.

Speed is another difference. You cannot get much speed out of ctypes, but
you can get full C speed out of Cython, by moving critical functionality
from Python down to Cython or C.

Besides that, be aware that in most cases, you usually do not need to "wrap
each C++ function", at least not in the beginning. Just wrap what you need,
when you need it. Writing the code that uses the wrapped library tends to
be a lot more work than the wrapping itself.

Stefan

[vybhav ramachandran]

Thank you, Stefan and Da-woods.

To simplify wrapping, I came across a project called Xdress but it doesn't seem to be maintained now. Do you know any other tips for easy wrapping using Cython?

Best,

Vybhav

[Stefan Behnel]

vybhav ramachandran schrieb am 12.11.20 um 11:26:

> To simplify wrapping, I came across a project called Xdress but it doesn't
> seem to be maintained now. Do you know any other tips for easy wrapping
> using Cython?

https://github.com/cython/cython/wiki/AutoPxd

Stefan

[vybhav ramachandran]

Thank you!

Python-autopxd2 looks promising with it's support for Python 3.6. However, it dosn't say anything about C++ support. Can I assume that it only support wrapping C Code?

Best,
Vybhav

[mattip]

There are a few more alternatives:

- pybind11 (already mentioned)

- cffi https://cffi.readthedocs.io/en/latest/

- cppyy https://cppyy.readthedocs.io/en/latest

CTypes, for good and for bad, is included in the stdlib, the others require installation. cppyy uses a llvm runtime library so its installation is heavy. The others are simple to install. When weighting the possibilities, three criteria stand out for me: ease of use, maintainability, and performance. Here are my (maybe wrong) opinions, corrections or other opinions welcome.

Ease of use

pybind11 and cppyy are designed for c++ code. cython also has some support for c++. All of them have a learning curve. As others mentioned, ctypes (and cffi) will require either refactoring your headers to use `extern "C"` or discovering the mangled name.

CTypes will require reflecting the C structures and functions in python (best practices means you should state the `argtypes` and `restype` of each function or use a `PYFUNCTYPE` to make sure your users are calling the functions properly), the others know something about the code by parsing the headers.

Cython is designed for integrating C and python code, so wrapping code comes naturally to it. CFFI requires some care when using pointers.

Maintainability

The packages that parse headers can deal gracefully with some API changes. You may need to change your python code that calls the library to reflect the changes. CTypes will require manually tracking API changes. Abstractions always leak, so any changes in the library you are wrapping may leak into your python code. Cython and pybind11 will require recompilation to handle new headers (beyond python-level code adjustments). Depending on how you use it, so will CFFI. CTypes is not compiled.

Performance

CTypes will be the slowest of all the solutions (as will the non-compiled CFFI mode). cffi and cppyy will provide faster interfaces in PyPy. Cython, pybind11, cffi and cppyy should be equivalent on CPython (when using the compiled CFFI mode), with a small advantage to cython and pytbind11 (depending on the use case).

I created a small tutorial in wrapping c code in CTypes, Cython, cffi and cppyy (I wrote it before pybind11 was a thing), you can play with the jupyter notebook https://github.com/mattip/c_from_python

Matti

[vybhav ramachandran]

Thank you so much!

Best,

Vybhav