lambda callable in nogil block

29 views
Skip to first unread message

Potato

unread,
Apr 17, 2022, 7:52:40 AMApr 17
to cython-users
I would like to pass a lambda function as argument into a nogil function and call it there. Is there any way to do that?
The simple
cdef object f = lambda int i: i
requires gil, I tried to ram 'nogil' in there but no placement worked, using the decorator as
cdef object f = cython.nogil(lambda int i: i)
does not work either.
C++ syntax is not supported and i wouldn't know what the type should be anyway.

I ended up creating a class for every usecase I needed and put them into fused types, which kind of worked, except
__call__ always requires gil and operator() can apparently be overloaded only in verbatim extern block,
making the code even more ugly.

And then I found out I can't nest these, as class attributes can't be fused types.
Please send help.

da-woods

unread,
Apr 17, 2022, 8:15:22 AMApr 17
to cython...@googlegroups.com
The basic issue is that a lambda function is a Python object so is not usable in a nogil environment.

You're probably better off using C function pointers (but they don't support capturing variables). But something like

cdef int return_i(int i) nogil:
    return i

cdef some_nogil_function(int (*func)(int) nogil) nogil:
  return func(10)

some_nogil_function(return_i)


However, you should also ask yourself: do you have a good reason to need nogil? Maybe you should just use a regular lambda function with the gil?
Please send help. --

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/85bd2068-dc36-44be-b1df-197af3af3391n%40googlegroups.com.


Potato

unread,
Apr 17, 2022, 10:14:27 AMApr 17
to cython-users
I guess I did not make it explicit -- I want to capture variables.
The code is evaluated in prange. I want to avoid the multiprocessing package and its shared arrays, if possible.

The other possible solution I forgot to mention is to pass a (cpp)class method pointer as an argument.
If I could create subclasses and overwrite the method, and a pointer to the subclass method had the same type,
that would avoid my problem with the fused types, but I don't know how to do that properly.

da-woods

unread,
Apr 17, 2022, 1:39:37 PMApr 17
to cython...@googlegroups.com
The classic C solution to this type of problem is to use a void* as an argument to your function pointer to capture the arguments. It'd typically point to some struct specific to the function. For example

cdef struct add_value_struct:
  int value

cdef int add_value(int x, void* s) nogil:
  cdef add_value_struct* s_cast = <add_value_struct*>s
  return x+s_cast.value

cdef some_nogil_function(int (*func)(int, void*) nogil, void* closure) nogil:
  return func(10, closure)

cdef add_value_struct s = {'value': 5}
some_nogil_function(add_value, <void*>(&s))


Alternatively you might be able to use inheritance and cdef classes (instead of fused types):

cdef class Base:
  cdef int call(self, int x) nogil:
      return 0  # dummy implementation

cdef class Impl1(Base):
  cdef int value
  cdef int call(self, int x) nogil:
    return x+self.value

cdef class Impl2(Base):
  ...

cdef some_nogil_function(Base f) nogil:
  return f.call(10)

You may run into the problem that you're a bit restricted in what you can do in cdef classes without the GIL - they are ultimately Python objects so anything that causes reference counting will need to GIL for example. But something like that should work.

However, it's sometimes a choice between Python-level convenience and C-level speed. You can't always have both (easily)

Potato

unread,
Apr 18, 2022, 6:11:45 AMApr 18
to cython-users
The first part is what I have done, with the added fused types.
I guess I should have provided some example code to make this easier.
The inheritance seems to work for me. Thank you.
Reply all
Reply to author
Forward
0 new messages