cffi and non-reentrant C libraries

[lgau...@gmail.com]

Hi,

Is there guidance about calling non-reentrant C libraries through cffi since the GIL is released? To my knowledge no way to tell cffi that a C call should be made without releasing the GIL.

At the moment I am using a `threading.RLock` context to make calls. Does it seem sound?

In addition to the RLock, I am unsure about how many backdoors through C allow an unwanted concurrent use of non-reentrant C functions. One I am thinking about is through a level of recursion: a C function calling an embedded Python itself calling that C function through cffi. I am not sure about how to best address that. Does anyone have recommendations?

Best,


Laurent

[Armin Rigo]

Hi,

On Sat, 4 Mar 2023 at 18:34, lgau...@gmail.com <lgau...@gmail.com> wrote:
> Is there guidance about calling non-reentrant C libraries through cffi since the GIL is released? To my knowledge no way to tell cffi that a C call should be made without releasing the GIL.
>
> At the moment I am using a `threading.RLock` context to make calls. Does it seem sound?

Yes, that's the right approach. In CFFI the answer is often: "what
would you do in C? Do the same thing." If you want to use a library
that is not thread-safe inside a wider context where threads exist,
the common solution is to wrap it manually with locks.

> In addition to the RLock, I am unsure about how many backdoors through C allow an unwanted concurrent use of non-reentrant C functions. One I am thinking about is through a level of recursion: a C function calling an embedded Python itself calling that C function through cffi. I am not sure about how to best address that. Does anyone have recommendations?

Depending on the library, you may want to wrap it using
``_thread.allocate_lock()`` or the equivalent from ``threading``,
instead of ``RLock``. This would deadlock if the Python programmer
does something like what you are describing, but that's a lot better
than crashing, and typing Ctrl-C should give a nice traceback. You
can also use more advanced forms of locks that detect deadlock
attempts and complain.


A bientôt,

Armin

[Laurent Gautier]

Thanks Armin, this is very helpful.

--
-- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "python-cffi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-cffi/n9pi8VILSYQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-cffi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-cffi/CAMSv6X0jwYPYVG_vC-P5usO389c-Z23kQ%3DM%2B6CBZAidwYCmi4w%40mail.gmail.com.