integrating a C package with a Python package

[Juan]

I have a fundamental question regarding the possibility of creating a hybrid dynamical model using python and C via cython, cffi or ctypes. 

There is a complicated fluid dynamical solver written in C developed by me and one of my colleagues has developed a machine learning package in python. At the moment, we are trying to integrate these two codes into a comprehensive package in python or C.

I found quite a few approaches to calling C functions from python via cython, ctypes or cffi and had successfully implemented two approaches myself to call C functions from python. However, none of these approaches really solved my problem. 

The idea is that we train the AI system in Python on-fly by feeding the solutions of the fluid dynamical system in sequence as we integrate it in time. Due to the large volume of the solutions for the fluid problem, we cannot save them on the hard disk. Hence, we plan to design a Python or C package which constantly alternates the calls between Python and C routines. However, technically, it seems to be a huge challenge for us. Because before doing the numerical computation of my fluid dynamical solver, a few very large arrays must be allocated, such as the memory space for storing the inertial term and MPI buffers. Till now I still do not know the feasible/easy way of how to allocate these arrays in C and reuse them for computation at the next time step after calling the AI training routines in Python. Please bear in mind that for our application, memory management is a major challenge. 

Could anyone give us suggestions? 

Perhaps, a better way to do this is to declare and manage all arrays in Python and pass the reference of the allocated memory to C function for computation. My major concern is to avoid constantly copying a big chunk of data between C and Python. For plan B, is the ctypes, cffi or cython preferred? 

My knowledge of Python is really limited. I appreciate that if you could point me to a specific example for learning. Many thanks in advance!!!!

Juan

[D Woods]

> Perhaps, a better way to do this is to declare and manage all arrays in Python and pass the reference of the allocated memory to C function for computation. My major concern is to avoid constantly copying a big chunk of data between C and Python.

If you use something like Numpy arrays and the interact with them in Cython using typed memoryviews you aren't actually copying data - you just have a view where you can write into your existing array data. You can also get pointers into these arrays if that helps.

Another alternative might be to write a cdef class in Cython that wraps your C data and provides the interface to interact with it from Python (see https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html#declaring-a-c-class-interface, but it isn't just restricted to wrapping C++ classes). Ownership is through the cdef class, which holds the underlying C structures, so there isn't much copying here either.

> For plan B, is the ctypes, cffi or cython preferred?

Not sure what "plan B" is but: the chances are you could do what you want in all three libraries. The advantage of ctypes/cffi is they don't require you to compile anything, so you can use them to interact with existing libraries directly from Python with no preparation. The advantage of Cython is that it can give you really seamless mixing of Python and C/C++ code. I'd probably recommend Cython, but you are asking on the Cython-user''s mailing list...

[sikik...@gmail.com]

Hello, 

https://github.com/ludlows/PESQ 

here is an example wrapping C functions using Cython. 

you can follow this project structure and have an intuition about this kind of projects.

hope it helps 

[Juan]

Thank you so much for both of you, D Wood and Sikik for your help!

I think I start to understand the problem now. If I want to access the C package on the Python side, I have to move the main() function from the C code to a .pyx file in Cython, in which all variables are defined. The memory allocation of these variables can be carried by the C routines and remains in the programe. The actual locations of these variables are shared by C and python. It means that I do not have to reallocate or assign the values to these variables again when I call the C functions next time.