Yet another concurency problem...

[Jérôme Kieffer]

Dear Cythoners,

I am wondering how to limit the number of (python-)threads concurently running in a nogil section of a "cdef class":

In python I would use a semaphore from threading. Is the openmp items like:
omp_get_num_threads/omp_get_max_threads working in this context ?


cdef class processing:
#I need a C/C++ like semaphore
cdef Semaphore sem
def __cinit__(self):
self.sem=Semaphore(8)
def bigCalc(self, float[:,:] img not None):
with self.sem:
with nogil:
process(img)

thanks for your ideas. Cheers.
--
Jérôme Kieffer <goo...@terre-adelie.org>

[Stefan Behnel]

Jérôme Kieffer, 04.10.2012 22:00:

> Dear Cythoners,
>
> I am wondering how to limit the number of (python-)threads concurently running in a nogil section of a "cdef class":
>
> In python I would use a semaphore from threading. Is the openmp items like:
> omp_get_num_threads/omp_get_max_threads working in this context ?

I don't think they'd help. They only give you the current number of
available threads, and they only relate to OpenMP threads, not general
Python threads.

How do you create the threads? Wouldn't a parallel() block allow you to
create the number of threads you need?

> cdef class processing:
> #I need a C/C++ like semaphore
> cdef Semaphore sem
> def __cinit__(self):
> self.sem=Semaphore(8)
> def bigCalc(self, float[:,:] img not None):
> with self.sem:
> with nogil:
> process(img)

That approach looks generally ok to me.

Stefan

[Nathaniel Smith]

Perhaps you can just use a python semaphore, and acquire/ release it just outside of the nogil section?

[Jérôme Kieffer]

On Sat, 6 Oct 2012 13:22:44 +0100
Nathaniel Smith <n...@pobox.com> wrote:

> Perhaps you can just use a python semaphore, and acquire/ release it just
> outside of the nogil section?

I think it is a bit more complicated than that ... it needs to be an official cython type because it is a attribute in a cdef class.
I started translating the python threading Semaphore on top of the "FastRLock" from Stefan (Thanks by the way:) and it seems to be working but I have others problems like:

How can one PYX file call a class from another PYX ?

Once this is solved. I will try to realease this cy_threading :)

Cheers,

--
Jérôme Kieffer <goo...@terre-adelie.org>

[Stefan Behnel]

Jérôme Kieffer, 06.10.2012 22:16:

> On Sat, 6 Oct 2012 13:22:44 +0100
> Nathaniel Smith wrote:
>> Perhaps you can just use a python semaphore, and acquire/ release it just
>> outside of the nogil section?
>
> I think it is a bit more complicated than that ... it needs to be an official cython type because it is a attribute in a cdef class.

You can type those as "object" if you don't want to be specific about the
Python type. Actually, if you just write "cdef somename", then "somename"
will be declared as plain object, as that is the default type (same as for
function arguments).

> I started translating the python threading Semaphore on top of the "FastRLock" from Stefan (Thanks by the way:)

Yep, I needed this for Lupa at some point and thought that it would be a
good idea to share it separately.

> and it seems to be working but I have others problems like:
> How can one PYX file call a class from another PYX ?

Two modules can cimport from each other. There are a couple of ways how
this can be done and not all of them are (well) documented. Here are the
basics:

http://docs.cython.org/src/userguide/sharing_declarations.html

And here is an example for public C-APIs exported by a module (each
separator marks the beginning of the file it names):

https://github.com/cython/cython/blob/master/tests/build/module_api.srctree

Note the "api" classifier that lets Cython generate a .pxd file and a .h
file that describes the exported functions (those having an "api"
classifier) and types (all defined cdef classes). You can then cimport from
the .pxd file and Cython will make it 'magically' work.

> Once this is solved. I will try to realease this cy_threading :)

Sure, just post it somewhere so that others can take a look and comment.

Stefan