cython.parallel and pure Python mode

[Matěj Laitl]

Hi again,
sorry to be recurring a bit, but I've came across another set of issues, now dealing with parallelisation in pure Python mode.

1. "with nogil:", "with gil:" context managers seem only to be available in Cython space. It would be great if one could just "from cython.parallel cimport gil, nogil" in .pxd and then use them in .py. With a trick and no-op Python implementation this could be purest Python combatible. (purest Python == when your .py files work even without cython installed)

2. "from cython.parallel cimport prange" does nothing, only "from cython.parallel import prange" works AFAICS. This breaks purest Python mode and my assumption that cimport affects generated C code, and import affects mainly runtime.

3. Declaring a method as "with gil" propagates to subclasses and cannot be undone (i.e. made plain nogil) in subclasses AFAICS. My view is that "with gil" is a function implementation detail and only the implicit nogil should be part of the signature. Note that this cannot be currently worked-around by putting the method body under "with gil:" in all cases, see "List-like container for Extension types and getting an item with nogil" thread. Alternatively, this could be solved by allowing Python locals in nogil functions provided they are accessed only under "with gil:".

4. (Bonus point) It would be nice if cpdef functions could be declared nogil, which would mean "the C function is nogil", the Python wrapper would still require the GIL and would drop it just to execute the C function.

5. (Bonus point) It would be nice if "normal" cdef methods could be overriden by nogil methods in subclasses. Not sure whether technically possible, as it seems that a function needs to know whether it is executed with/without GIL at C compile time. Low priority, can be worked-around easily.

While I understand none of the above is a real show-stopper, at least 1-3 would help us (scientific research of atmospheric dispersion model assimilation) immensely. I'm willing to contribute, I can for sure write test cases for the above or try to implement it in longer-term if it gets status "nice to have, but nobody to implement it currently".

Regards,

               Matěj

[Matěj Laitl]

On Sunday, August 26, 2012 2:37:24 PM UTC+2, Matěj Laitl wrote:

4. (Bonus point) It would be nice if cpdef functions could be declared nogil, which would mean "the C function is nogil", the Python wrapper would still require the GIL and would drop it just to execute the C function.

Oh, this seems to be already possible, my fault.