lambda callable in nogil block

[Potato]

I would like to pass a lambda function as argument into a nogil function and call it there. Is there any way to do that?
The simple
cdef object f = lambda int i: i
requires gil, I tried to ram 'nogil' in there but no placement worked, using the decorator as
cdef object f = cython.nogil(lambda int i: i)
does not work either.
C++ syntax is not supported and i wouldn't know what the type should be anyway.

I ended up creating a class for every usecase I needed and put them into fused types, which kind of worked, except
__call__ always requires gil and operator() can apparently be overloaded only in verbatim extern block,
making the code even more ugly.

And then I found out I can't nest these, as class attributes can't be fused types.
Please send help.

[da-woods]

The basic issue is that a lambda function is a Python object so is not usable in a nogil environment.

You're probably better off using C function pointers (but they don't support capturing variables). But something like

cdef int return_i(int i) nogil:

    return i

cdef some_nogil_function(int (*func)(int) nogil) nogil:

  return func(10)

some_nogil_function(return_i)

However, you should also ask yourself: do you have a good reason to need nogil? Maybe you should just use a regular lambda function with the gil?

Please send help. --

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/85bd2068-dc36-44be-b1df-197af3af3391n%40googlegroups.com.

[Potato]

I guess I did not make it explicit -- I want to capture variables.
The code is evaluated in prange. I want to avoid the multiprocessing package and its shared arrays, if possible.

The other possible solution I forgot to mention is to pass a (cpp)class method pointer as an argument.
If I could create subclasses and overwrite the method, and a pointer to the subclass method had the same type,
that would avoid my problem with the fused types, but I don't know how to do that properly.

[da-woods]

The classic C solution to this type of problem is to use a void* as an argument to your function pointer to capture the arguments. It'd typically point to some struct specific to the function. For example

cdef struct add_value_struct:

  int value

cdef int add_value(int x, void* s) nogil:

  cdef add_value_struct* s_cast = <add_value_struct*>s

  return x+s_cast.value

cdef some_nogil_function(int (*func)(int, void*) nogil, void* closure) nogil:

  return func(10, closure)

cdef add_value_struct s = {'value': 5}

some_nogil_function(add_value, <void*>(&s))

Alternatively you might be able to use inheritance and cdef classes (instead of fused types):

cdef class Base:

  cdef int call(self, int x) nogil:

      return 0  # dummy implementation

cdef class Impl1(Base):

  cdef int value

  cdef int call(self, int x) nogil:

    return x+self.value

cdef class Impl2(Base):

  ...

cdef some_nogil_function(Base f) nogil:

  return f.call(10)

You may run into the problem that you're a bit restricted in what you can do in cdef classes without the GIL - they are ultimately Python objects so anything that causes reference counting will need to GIL for example. But something like that should work.

However, it's sometimes a choice between Python-level convenience and C-level speed. You can't always have both (easily)

To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/197028be-5ed9-4658-9477-728092963f3dn%40googlegroups.com.

[Potato]

The first part is what I have done, with the added fused types.
I guess I should have provided some example code to make this easier.
The inheritance seems to work for me. Thank you.