Calling gil-requiring function not allowed without gil
Ting Zhou
here is my pyx file. When I compile it, it says "Calling gil-requiring
function not allowed without gil" at calling dabs(x[i]). Why my dabs
function is a gil-requiring function?
====================================
from cython.parallel import *
import numpy as np
cimport numpy as np
cimport cython
cdef double dabs(double x):
if x < 0: return -x
else: return x
@cython.boundscheck(False)
def func(np.ndarray[np.float64_t, ndim=1] x):
cdef int i, len_x=x.shape[0]
cdef double sum=0.0
with nogil, parallel(2):
for i in prange(len_x):
sum+=dabs(x[i])
return sum
====================================
best wishes,
ting
Dag Sverre Seljebotn
> Hi,
> here is my pyx file. When I compile it, it says "Calling gil-requiring
> function not allowed without gil" at calling dabs(x[i]). Why my dabs
> function is a gil-requiring function?
>
> ====================================
> from cython.parallel import *
> import numpy as np
> cimport numpy as np
> cimport cython
>
> cdef double dabs(double x):
This should be
cdef double dabs(double x) nogil:
DS
Stefan Behnel
> On 08/11/2011 01:43 PM, Ting Zhou wrote:
>> here is my pyx file. When I compile it, it says "Calling gil-requiring
>> function not allowed without gil" at calling dabs(x[i]). Why my dabs
>> function is a gil-requiring function?
>>
>> ====================================
>> from cython.parallel import *
>> import numpy as np
>> cimport numpy as np
>> cimport cython
>>
>> cdef double dabs(double x):
>
> This should be
>
> cdef double dabs(double x) nogil:
Note that Cython cannot infer this automatically. Even a trivial function
may require an exclusive global lock for some reason, and it's common to
use the GIL for that. So the programmer must be explicit here.
Stefan
coomteng
Dag Sverre Seljebotn
Are you still against this mini-CEP?:
with cython.global_lock():
...
Where global_lock() is GIL-requiring noop.
This
a) Improves code readability vastly. Having a critical section take
effect because of the *lack* of a keyword is just very odd to anyone
who's not shoulder deep in CPython internals
b) Allows inferring 'nogil' (= lower learning curve, one keyword less
to learn!)
c) Allows Cython for .NET and friends to take advantage of
multi-threading eventually.
Anyway, I don't have time for a flamewar about this right now, so I'm
just throwing it out there to hear your opinion.
Dag Sverre
Robert Bradshaw
<d.s.se...@astro.uio.no> wrote:
> On 08/11/2011 02:13 PM, Stefan Behnel wrote:
>>
>> Dag Sverre Seljebotn, 11.08.2011 13:58:
>>>
>>> On 08/11/2011 01:43 PM, Ting Zhou wrote:
>>>>
>>>> here is my pyx file. When I compile it, it says "Calling gil-requiring
>>>> function not allowed without gil" at calling dabs(x[i]). Why my dabs
>>>> function is a gil-requiring function?
>>>>
>>>> ====================================
>>>> from cython.parallel import *
>>>> import numpy as np
>>>> cimport numpy as np
>>>> cimport cython
>>>>
>>>> cdef double dabs(double x):
>>>
>>> This should be
>>>
>>> cdef double dabs(double x) nogil:
>>
>> Note that Cython cannot infer this automatically. Even a trivial
>> function may require an exclusive global lock for some reason, and it's
>> common to use the GIL for that. So the programmer must be explicit here.
>
> Are you still against this mini-CEP?:
>
> with cython.global_lock():
> ...
>
> Where global_lock() is GIL-requiring noop.
Just reading this, it's not immediately obvious what this means. (I
thought at first this was different syntax for "with gil"...)
> This
>
> a) Improves code readability vastly. Having a critical section take effect
> because of the *lack* of a keyword is just very odd to anyone who's not
> shoulder deep in CPython internals
I'm not following you here. The only way to run into this is if you
have explicitly release it. Presumably you can learn both keywords at
the same time. Perhaps extern function could be nogil by default.
> b) Allows inferring 'nogil' (= lower learning curve, one keyword less to
> learn!)
We can't just change the meaning of existing code, not the unsafe
direction at least. The more inferring the better though, and I think
it might be useful to move to a place where we assume the GIL need not
be held unless specifically requested or needed for Python operations.
Dag Sverre Seljebotn
True. "cython.synchronized"? The synchronized keyword in Java does not
quite the same thing but almost.
>> This
>>
>> a) Improves code readability vastly. Having a critical section take effect
>> because of the *lack* of a keyword is just very odd to anyone who's not
>> shoulder deep in CPython internals
>
> I'm not following you here. The only way to run into this is if you
> have explicitly release it. Presumably you can learn both keywords at
> the same time. Perhaps extern function could be nogil by default.
I'll try to explain again. Consider code like this:
cdef call_c():
magic_c_function(3)
This code is perfectly fine even if magic_c_function is not reentrant --
but that's hardly well documented! So it's conceivable that another
programmer comes along (who doesn't know the C library well) and decides
that "somebody has just been too lazy to add the nogil specifier" and
slaps it on -- at which point you have a bug.
This is more to the point in creating readable code IMO:
cdef call_c():
with cython.synchronized():
magic_c_function(3)
Also, consider your classical race conditions:
cdef init():
global resource
if resource == NULL:
resource = malloc(sizeof(resource_t))
...
This is safe code. But in pure Python code, the similar logic would be
an unsafe race condition if you used threading. So while Python requires
explicit locking, while Cython requires explicit "unlocking", and
there's a bit of an asymmetry there.
>> b) Allows inferring 'nogil' (= lower learning curve, one keyword less to
>> learn!)
>
> We can't just change the meaning of existing code, not the unsafe
> direction at least. The more inferring the better though, and I think
> it might be useful to move to a place where we assume the GIL need not
> be held unless specifically requested or needed for Python operations.
Well, moving to such a place would be changing the meaning of existing
code. But it seems like we're on the same page, what you say in the last
sentence is exactly what I want. Introducing "cython.synchronized" would
simply be a way of starting the (year-long) deprecation cycle. It'd be
straightforward to issue warnings on all external code that isn't
explicitly declared "synchronized" or "nogil" and so on.
I guess another important point is
d) It'd be nice to have .NET threading semantics in Cython for
IronPython, or Java threading semantics in Cython for Jython, or to
gracefully support any GIL-less fork of Python.
Dag Sverre
Stefan Behnel
Dag Sverre Seljebotn, 12.08.2011 08:50:
I think I'm as confused as Robert here. Is that the GIL or some separate lock?
If the latter, where would that lock be stored? Module-wide? While there
are certainly use cases for that, I think it's much more common to either
a) use the GIL or b) use an explicit lock at some well defined point, e.g.
at an object or class level.
I don't think we disagree when I say that locking should be explicit, but
that includes the "thing" that keeps the lock during its lifetime.
> Also, consider your classical race conditions:
>
> cdef init():
> global resource
> if resource == NULL:
> resource = malloc(sizeof(resource_t)) ...
>
> This is safe code.
Well, it's safe code as long as there is no Python code interaction in
between. Calling back into the interpreter may trigger a thread switch.
Admittedly, this is not obvious and tends to introduce bugs. I learned that
the hard way in lxml, actually twice.
If your "synchronised" directive refers to the GIL, then it would suffer
from that problem as well. I think that's very undesirable for an explicit
critical section statement.
Stefan