Both ABI and API modes with the same codebase?

[Matthias Geier]

Dear list.

Sorry for the vague question, but probably someone has already done
this and I don't have to start from scratch ...

I have written a wrapper
(https://github.com/spatialaudio/python-sounddevice) for a C library
(PortAudio) using CFFI with its in-line ABI mode. This works great and
I definitely want to keep maintaining that.

However, there is a certain use case, where using the API mode would
be preferable. Namely, the C library uses a callback function that is
called in a high-priority thread and for some specialized
applications, it is supposed to do some work in a fixed time frame of
a few milliseconds. Implementing this callback in Python is not
reliable enough because of garbage collection and the GIL which is
shared between the high-priority thread and the rest of the Python
program.

I think it would make sense to implement the callback in C in this
case and provide the code in ffibuilder.set_source(). This way, the
callback could run in native C speed and without using the GIL.

Now my questions:

Can I have both within the same codebase?

Has anybody already tried a similar thing?

Is this a bad idea?


cheers,
Matthias

[Daniel Holth]

Do you intend for the library user to provide their own C callback, passing it to the extension, or will you provide the implementation?

It sounds like you like not requiring a compiler in most cases, but view ffibuilder.set_source() as a convenient compiler interface. What if you thought about the problem as building a version of PortAudio with one additional function and using the larger library in ABI mode, or as using a second, one-function library, in ABI mode?

Once you have the shared library, you will need to get a function pointer to your callback by any means (even the primitive dlopen call would be enough) and then cast it to a PortAudio-in-ABI-mode function pointer for the inevitable 'set callback' function. ffi.addressof(lib, "name") ?

For example, I will sometimes abuse distutils' Extension() to build shared libraries that are not CPython extensions, and then open the libraries with ctypes or cffi.

A different way to build shared libraries, by embedding the tinycc compiler in Python: https://pypi.python.org/pypi/tinycc 

[Matthias Geier]

The former. But this would be a very specialized use case. The majority of users would use Python in their callbacks.
 

It sounds like you like not requiring a compiler in most cases, but view ffibuilder.set_source() as a convenient compiler interface. What if you thought about the problem as building a version of PortAudio with one additional function and using the larger library in ABI mode, or as using a second, one-function library, in ABI mode?

The idea with a second library sounds very promising, I didn't think of that!

The only change to my original ABI-mode wrapper would be to enable passing a C callback function pointer instead of a Python callable.

This sounds doable, I'll try that!
 

Once you have the shared library, you will need to get a function pointer to your callback by any means (even the primitive dlopen call would be enough) and then cast it to a PortAudio-in-ABI-mode function pointer for the inevitable 'set callback' function. ffi.addressof(lib, "name") ?

The PortAudio function Pa_OpenStream() simply takes a pointer to the callback function.
If ffi.addressof() provides that, I should be all set!

For example, I will sometimes abuse distutils' Extension() to build shared libraries that are not CPython extensions, and then open the libraries with ctypes or cffi.

OK, sounds interesting. But what would be the advantage over using CFFI for compiling?
 

A different way to build shared libraries, by embedding the tinycc compiler in Python: https://pypi.python.org/pypi/tinycc 

This looks like being Windows only. I'd like to have a more platform-independent solution.

Anyway, I think I have enough information to get started now, thanks a lot for your help!

cheers,
Matthias