Best way to pass functions pointers alongside Python functions

[Max Bachmann]

In one of my projects I provide the following functionality:

```

def metric(a, b):

      ...

      return result

def apply(list_a, list_b, metric):

    for a in list_a:

        for b in list_b:

            metric(a, b)

```

since these constant type conversions become very slow and in many cases more optimized algorithms can be used when working on multiple data elements I match the passed metric to check, whether I can directly use a C/C++ implementation:

```

def c_func(metric):

    if metric is metric1:

        # use c imple of metric 1

    if metric is metricN:

        # use c imple of metric 2

   else

       # call Python function

```

This is a lot faster, but has a couple of disadvantages:

1) the apply function needs to be updated when new metrics are added

2) it does not allow other modules to provide efficient metrics that the apply function does not know

Now I am searching for a better way to pass these functions pointers alongside the functions. So far I had the following ideas:

1) replace the functions with extension types using __call__ for the normal function call

```

cdef class Metric:

    cdef functptr

     @staticmethod

     def __call__(a, b):

        return result

metric = Metric()

```

2) add an extension types into the dict of the metric function

```

def metric(a, b):

    return result

cdef MetricCallback cb = MetricCallback()

cb.funcptr = funcptr

metric.__ModuleName_MetricCallback = cb

```

Currently I am leaning towards the second version for the following reasons:

1) it is probably simpler to add in a third party library, since it does not change the type of the metric function

2) if in the future I decide, that for other functions more callbacks are needed I can simply add more objects into the dict without breaking backwards compatibility

Are there any advantages/disadvantages of these approaches I am missing? Are there better ways to achieve this, or even other implementations already making use of similar concepts?

[Golden Rockefeller]

If your apply method is in raw (not compiled) python code, and you want to expose a c function with cython: then I would recommend wrapping the c function in a python function:

# apply_code.py

def apply(list_a, list_b, metric):

    for a in list_a:

        for b in list_b:

            metric(a, b)

-----------------------------------------

# exposing_code.pyx

# ... extern c_metric here ...

def exposed_metric(a, b):

    c_metric(a, b) # you may need to add some conversion code for a and b if necessary

----------------------------------------

# main.py

from exposing_code import exposed_metric

from apply_code import apply

if __name__=

    a = [1, 2,3]

    b = [7, 2, 6]

    apply(a, b, exposed_metric)

If your apply method is in cython code: then maybe(?) you can create a BoxedMetric extension (cdef) class that store the function ptr, and call that pointer directly. Something like this:

# apply_code.pxd

# Use a typedef on function pointer for convenience


ctypedef void (*metric_fn)(double, double) 





cdef class BoxedMetric:

    cdef metric_fn my_metric

------------------------------------

# apply_code.pyx

cdef class BoxedMetric:

    def __cinit__(self):

        my_metric = NULL

def apply(list_a, list_b, metric):

    if isinstance(metric, BoxedMetric):

       if metric.my_metric != NULL:

            for a in list_a:

                 for b in list_b:

                      metric.my_metric(a, b)

   else:

      for a in list_a:

          for b in list_b:

              metric(a, b)

-----------------------------------------

# exposing_code.pyx

from apply_code cimport BoxedMetric

# ... extern c_metric here ...

exposed_metric = BoxedMetric()

exposed_metric.my_metric = c_metric

----------------------------------------------

# main.py

from exposing_code import exposed_metric

from apply_code import apply

if __name__=

    a = [1, 2,3]

    b = [7, 2, 6]

    apply(a, b, exposed_metric)

   

The first method that I suggest here is simpler in my opinion. I am not that sure how smooth it is to get second recommendation to work (if at all). In isolation, one can assume that working with function pointers directly (second recommendation) can be than faster than using the calling python (def) functions (first recommendation) even if the python function is compiled. However, this is only an assumption without considering how and when the functions is called and what type the function returns. In your case, I don't know which recommendation will be faster or whether the difference is significant enough to matter.

Golden Rockefeller

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/13e5135d-8c93-4014-952f-591bfde2578fn%40googlegroups.com.

[D Woods]

A couple of comments:

1) __call__ shouldn't be a staticmethod. I'm not sure quick how Python will interpret this but it's likely to cause some confusion.

2) Scipy has a feature called "LowLevelCallable" which is designed to solve a similar problem. It lets you pass regular Python function, Cython functions, ctypes function pointers, Numba functions, and maybe a few other things.

David

[Max Bachmann]

Thanks for the input.

>  Scipy has a feature called "LowLevelCallable"

I did not know about this. I like this solution.

Max