Pickling cdef-class member c function pointer

[Zhang David]

I'm trying to seek ways to properly serialize a cdef-class w/ c function pointer as a member, by implementing custom __getstate__ / __setstate__ / __reduce__

An example could be:

ctypedef void (*CyNoArgObjMethod_t)(Service)

cdef class CyTask(Task):
    @staticmethod
    cdef CyTask init(Service s, void* cy_noarg_method):
        cdef CyTask task
        task = CyTask.__new__(CyTask)
        task.s = s
        task.cy_noarg_method = <CyNoArgObjMethod_t>cy_noarg_method
        return task

    cdef void run(self) except *:
        self.cy_noarg_method(self.s)

I've tried converting into bytes and vice versa but it doesn't seem to work. 

Reason for doing this is to maintain the static type of this functor to speed things up. 

At the same time, I want to be able to pickle this to allow things like checkpointing. 

Any idea/suggestions how I could achieve this? Thanks!

[da-woods]

If it's an arbitrary function pointer, and you're aiming to deserialize it on a different machine then you definitely can't do it. Simply because the function it points to may not even exist on that machine. And it if it does then it's almost certainly at a different address.

If you have a limited static list of functions that it could be, then use a lookup table of strings-to-function-pointers/function-pointers-to-string.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/2b2f2469-f1b4-4fd8-ad2b-7376aea4a9d9n%40googlegroups.com.

[Zhang David]

Thank you David for the quick response! 

The creation of this is actually to deal w/ this: 

https://github.com/cython/cython/issues/5062

i.e I want to enjoy the benefit of a static typed “c bounded function” (although there’s no such thing). 

The actual value passed in will be any Service or derived class of Service object’s function with no arg (so meets the functor type), so arbitrary I guess. 

Then my question is - is the any other workaround w/o c functor (the current way) that I can preserve the static type of this cdef “member func” w/o losing the signature entirely (i.e. object), while being able to pickle/serialize? 

To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/88a0243c-8dde-99da-c9ab-d23f8b330461%40d-woods.co.uk.

[da-woods]

The creation of this is actually to deal w/ this: 

https://github.com/cython/cython/issues/5062

i.e I want to enjoy the benefit of a static typed “c bounded function” (although there’s no such thing).

I think you're misunderstanding what that issue will actually do. Creating a "bound C function" creates a Python wrapper (both with and without the improvement proposed there). If you're calling your bound C functions that way then you should probably just write a regular "def" function because there's actually no difference (except that you've also imposed all the limitations of a cdef function). That issue proposes binding them in a slightly quicker way but doesn't change that much.

The actual value passed in will be any Service or derived class of Service object’s function with no arg (so meets the functor type), so arbitrary I guess.

That isn't directly possible in either in Cython or in C). You need to come up with some alternative mechanism where you store the function and the information needed to reproduce it together.

Then my question is - is the any other workaround w/o c functor (the current way) that I can preserve the static type of this cdef “member func” w/o losing the signature entirely (i.e. object), while being able to pickle/serialize?

My best suggestion is that you don't store C function pointers but instead you store cdef classes that all inherit from some common base:

```
cdef class Callable:
    cdef void call(self):  # or whatever types you actually have
        pass
```

You then create a bunch of derived types that inherit from callable, and are pickleable. Calling `callable_instance.call()` is quick (although probably not quite as quick as going through a function pointer). I don't know if that fits into your scheme.

To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/CAHcVAySfeGcOzEU5UpcCvSCv0AkQKwukiys0LE%3Dh9xTZzZH24A%40mail.gmail.com.

[Zhang David]

thank you David for the detailed explanation & the suggestion!

reason for me to come up w/ the functor idea was b/c I was too lazy impl all the derived classes lol... but back then i wasn't thinking about the pickling and stuff. 

thanks again - fully agree w/ the derived class idea. will move to that. 

Best,