with nogil
Eric Frederich
I'd like some advice. I have bindings Python bindings built using Cython for a 3rd party C library.
For each C call, I call it with the "with nogil" context processor.
I do this because some of the calls to this library may take some time and if I have my GUI on a separate thread using Python I don't want my GUI to lock up until the C call returns.
This has worked wonderfully except now I have come to a situation where one of those C calls without the GIL does something which causes some other Python code to run.
I get segfaults when this happens. I removed the "with nogil" to verify this was causing the problem and it did not crash without it.
I will need to remove the "with nogil" context processor by default but I'd like to be able to selectively use it on a case by case basis.
What would be a good way to enable this?
I was thinking about adding a kwarg parameter to each function call to allow or disallow it.
That might get a little messy.
I also thought about having a global boolean in the bindings themselves that controls whether to use nogil or not.
If I went with this option, I could write my own context processor called "nogil" which in turn sets this global variable in the bindings causing the "real" nogil to be used.
What are your thoughts.
Thanks,
~Eric
mark florisson
> Hello,
>
> I'd like some advice. I have bindings Python bindings built using Cython
> for a 3rd party C library.
> For each C call, I call it with the "with nogil" context processor.
> I do this because some of the calls to this library may take some time and
> if I have my GUI on a separate thread using Python I don't want my GUI to
> lock up until the C call returns.
>
> This has worked wonderfully except now I have come to a situation where one
> of those C calls without the GIL does something which causes some other
> Python code to run.
How exactly does that happen? Does it invoke a callback provided by
you? Do you have any control over what and how it executes this code?
Sturla Molden
>
> This has worked wonderfully except now I have come to a situation where one
> of those C calls without the GIL does something which causes some other
> Python code to run.
> I get segfaults when this happens. I removed the "with nogil" to verify
> this was causing the problem and it did not crash without it.
>
C code should never call back to Python without ensuring the GIL, e.g.
using the simplified GIL API. The GIL state need not be known in
advance. That is, it grabs the GIL and then returns the GIL state to
whatever it was before grabbing it. So you have a bug in the C code.
Make sure that whoever writes the C code knows to use the simplified C
API when calling back to Python. If they can use the Python C API to
call back to Python, they should know to ensure the GIL as well.
Sturla
Stefan Behnel
> I'd like some advice. I have bindings Python bindings built using Cython
> for a 3rd party C library.
> For each C call, I call it with the "with nogil" context processor.
> I do this because some of the calls to this library may take some time and
> if I have my GUI on a separate thread using Python I don't want my GUI to
> lock up until the C call returns.
>
> This has worked wonderfully except now I have come to a situation where one
> of those C calls without the GIL does something which causes some other
> Python code to run.
> I get segfaults when this happens.
In case you can control the place where that "Python code is caused to be
run", your code may just be missing a "with gil" when entering back into
your own code. If you have a callback function somewhere, you can add this
declaration to its signature to let Cython reacquire the GIL on entry, e.g.
cdef int mycallback(void* context) with gil:
...
Only if your 3rd party C library calls into Python directly (which is
rather unlikely), you're out of luck and must hold the GIL while calling
into it.
Stefan
Sturla Molden
> In case you can control the place where that "Python code is caused to be
> run", your code may just be missing a "with gil" when entering back into
> your own code. If you have a callback function somewhere, you can add this
> declaration to its signature to let Cython reacquire the GIL on entry, e.g.
>
> cdef int mycallback(void* context) with gil:
Yes.
> Only if your 3rd party C library calls into Python directly (which is
> rather unlikely), you're out of luck and must hold the GIL while calling
> into it.
Or tell who ever wrote it to do this:
{
/* ensure we hold the GIL */
PyGILState_STATE gilstate = PyGILState_Ensure();
/* Use the Python API */
/* restore the state of the GIL */
PyGILState_Release(gilstate);
}
Sturla