passing a pointer function to another function which can be called in python

[Zahra Sheikh]

I am trying to find a way to pass a pointer function as an argument to a python function similar to the following example

from cpython cimport array
import cython
import ctypes
cimport numpy as np

ctypedef void (*FuncDef)(double *, double *)
from cython.parallel import prange
cdef void func(int* x, double* hx, void(*func)(double*, double*), int n):
      def int i
      for i from 0 <= i < n:
          func(&x[i], &hx[i])
      return

cpdef void normal(double* u, 
                  double* yu):
      yu=-u*u*0.5
      return

def foo(np.ndarray[ndim=1, dtype=np.float64_t] x,
        np.ndarray[ndim=1, dtype=np.float64_t] hx,
        FuncDef dist, int k):
    cdef np.ndarray[ndim=1, dtype=np.float64_t] sp
    cdef int num=len(x)
    cdef int j
    for j from 0 <= j <k:
        func(&x[0],&hx[0],dist, num)
        sp[j]=hx[0]
    return sp

 
Is there any way to do this? I'd like to be able to insert different functions as parameter dist to the foo function and the foo function can be callable in python. Any suggestion would be appreciated a lot!

Regards

[Nils Bruin]

In order to send something to a python function, you have to wrap it in a python object. There is a standard python object in the C-API for wrapping pointers:

https://docs.python.org/3.1/c-api/capsule.html

[Stefan Behnel]

Nils Bruin schrieb am 24.09.2017 um 21:18:
> On Sunday, September 24, 2017 at 5:19:27 AM UTC-7, Zahra Sheikh wrote:
>>
>> I am trying to find a way to pass a pointer function as an argument to a
>> python function similar to the following example
>>

>> from cpython cimport arrayimport cythonimport ctypes
>> cimport numpy as np
>>
>> ctypedef void (*FuncDef)(double *, double *)from cython.parallel import prange

>> cdef void func(int* x, double* hx, void(*func)(double*, double*), int n):
>> def int i
>> for i from 0 <= i < n:
>> func(&x[i], &hx[i])
>> return
>>
>> cpdef void normal(double* u,
>> double* yu):
>> yu=-u*u*0.5
>> return
>> def foo(np.ndarray[ndim=1, dtype=np.float64_t] x,
>> np.ndarray[ndim=1, dtype=np.float64_t] hx,
>> FuncDef dist, int k):
>> cdef np.ndarray[ndim=1, dtype=np.float64_t] sp
>> cdef int num=len(x)
>> cdef int j
>> for j from 0 <= j <k:
>> func(&x[0],&hx[0],dist, num)
>> sp[j]=hx[0]
>> return sp
>>
>>
>> Is there any way to do this? I'd like to be able to insert different

>> functions as parameter *dist* to the *foo *function and the foo function

>> can be callable in python. Any suggestion would be appreciated a lot!
>

> In order to send something to a python function, you have to wrap it in a
> python object. There is a standard python object in the C-API for wrapping
> pointers:
>
> https://docs.python.org/3.1/c-api/capsule.html

Although in this specific case of (a set of?) alternative C functions, I'd
rather use a dedicated cdef class that wraps a function pointer. Much
simpler, faster, and trivially type safe.

Stefan

[Zahra Sheikh]

Hi Stefan,

I tried to apply your suggestion as follows
 

from cpython cimport array
import cython

import ctypes
cimport numpy as np

from cython.parallel import prange
cdef void sample(int* x, double* hx, void(*func)(double*, double*), int n):

      def int i
      for i from 0 <= i < n:
          func(&x[i], &hx[i])
      return

cpdef void normal(double* u, 
                  double* yu):
      yu=-u*u*0.5
      return

cdef class foo:
    cdef void (*func)(double *, double *) 
    def __cinit__(self, 
                  np.ndarray[ndim=1, dtype=np.float64_t] x,
                  np.ndarray[ndim=1, dtype=np.float64_t] hx,
                  int k):
    self.func= func
   

 cdef np.ndarray[ndim=1, dtype=np.float64_t] sp
    cdef int num=len(x)
    cdef int j
    for j from 0 <= j <k:

        sample(&x[0],&hx[0],self.func, num)

        sp[j]=hx[0]
    return sp

But I got this error message:
 self.func= func
                        ^
------------------------------------------------------------

Cannot convert Python object to 'void (*)(double *, double *)'

How should I define the pointer function for the class that won\t raise error message and I can initialize it by normal function?

cheers

[Stefan Behnel]

Zahra Sheikh schrieb am 25.09.2017 um 11:44:
> from cpython cimport arrayimport cythonimport ctypes
> cimport numpy as np

> from cython.parallel import prange
> cdef void sample(int* x, double* hx, void(*func)(double*, double*), int n):
> def int i
> for i from 0 <= i < n:
> func(&x[i], &hx[i])
> return
>
> cpdef void normal(double* u,
> double* yu):
> yu=-u*u*0.5
> return
> cdef class foo:
> cdef void (*func)(double *, double *)
> def __cinit__(self,
> np.ndarray[ndim=1, dtype=np.float64_t] x,
> np.ndarray[ndim=1, dtype=np.float64_t] hx,
> int k):
> self.func= func
> cdef np.ndarray[ndim=1, dtype=np.float64_t] sp
> cdef int num=len(x)
> cdef int j
> for j from 0 <= j <k:
> sample(&x[0],&hx[0],self.func, num)
> sp[j]=hx[0]
> return sp
>

> *But I got this error message:*

> self.func= func
> ^
> ------------------------------------------------------------
>
> Cannot convert Python object to 'void (*)(double *, double *)'

"func" is an undefined (global) name here.

You probably want to set "self.func" from outside of the class, and you
should also want to move all that code out of the __cinit__() method.
Calling the function is independent from creating the wrapper object.

> How should I define the pointer function for the class that won\t raise
> error message and I can initialize it by normal function?

I guess that you are looking for something like this:

cdef class _SampleFunc:

cdef void (*func)(double *, double *)

normal_func = _SampleFunc()
(<_SampleFunc>normal_func).func = normal

def py_sample(np.ndarray[ndim=1, dtype=np.float64_t] x,
np.ndarray[ndim=1, dtype=np.float64_t] hx,
int k, _SampleFunc sample_func):
func = sample_func.func
assert func is not NULL
... # do your processing here.

You probably want to rename everything to export "sample" and "normal" at
the API level, so that users can use nice names and pass them around. I
sometimes prefix internal C stuff with and underscore in my own code.

Stefan

[Stefan Behnel]

Zahra Sheikh schrieb am 25.09.2017 um 11:44:

> from cpython cimport arrayimport cythonimport ctypes
> cimport numpy as np

> from cython.parallel import prange
> cdef void sample(int* x, double* hx, void(*func)(double*, double*), int n):
> def int i
> for i from 0 <= i < n:
> func(&x[i], &hx[i])
> return
>
> cpdef void normal(double* u,
> double* yu):
> yu=-u*u*0.5
> return
> cdef class foo:
> cdef void (*func)(double *, double *)
> def __cinit__(self,
> np.ndarray[ndim=1, dtype=np.float64_t] x,
> np.ndarray[ndim=1, dtype=np.float64_t] hx,
> int k):
> self.func= func
> cdef np.ndarray[ndim=1, dtype=np.float64_t] sp
> cdef int num=len(x)
> cdef int j
> for j from 0 <= j <k:
> sample(&x[0],&hx[0],self.func, num)
> sp[j]=hx[0]
> return sp
>

> *But I got this error message:*

> self.func= func
> ^
> ------------------------------------------------------------
>
> Cannot convert Python object to 'void (*)(double *, double *)'

"func" is an undefined (global) name here.

You probably want to set "self.func" from outside of the class, and you
should also want to move all that code out of the __cinit__() method.
Calling the function is independent from creating the wrapper object.

> How should I define the pointer function for the class that won\t raise
> error message and I can initialize it by normal function?

I guess that you are looking for something like this:

cdef class _SampleFunc:

cdef void (*func)(double *, double *)

normal_func = _SampleFunc()
(<_SampleFunc>normal_func).func = normal

def py_sample(np.ndarray[ndim=1, dtype=np.float64_t] x,
np.ndarray[ndim=1, dtype=np.float64_t] hx,

[Zahra Sheikh]

Thanks Stefan,

Well I tried to use your comment in my code "ars.pyx" (Line 605 to the end) and it gets compiled by the python example code "test.py" returns the following error

s=ars.py_ars(ns, m, emax, num, x, hx, hpx, normal)
TypeError: Argument 'sample_func' has incorrect type (expected ars._SampleFunc, got builtin_function_or_method)

can anybody suggest how it can be solved? Thanks in advance!
P.S. I would like to be able to instantiate the function by different probability distribution functions for the class myARS and function "normal" is just an example of how this function should look like.

[Nils Bruin]

With "from ars import myARS, normal", you import the *python* function. That object isn't of python type _SampleFunc and python wouldn't know how to convert it to one either.

You probably need a piece of code of the type

normal_wrapper = _SampleFunc()
normal_wrapper.func = normal

or perhaps (i'm not sure if cython will automatically put the "address of" operator in):

normal_wrapper = _SampleFunc()
normal_wrapper.func = &normal

Note that in "ars.pyx" you now basically have two objects: the C-level "normal" and the python-level wrapper "normal", thanks to the "cpdef". I'm not sure that in the statement above, cython will be able to figure out which one to use.

[Zahra Sheikh]

Thanks Nils for the suggestion but I would like to be able to set different functions for _SampleFunc() not only the 'normal' function which I gave an example. Is there any way to make it more general?

cheers,
Z.

[Nils Bruin]

On Tuesday, September 26, 2017 at 7:22:16 AM UTC-7, Zahra Sheikh wrote:

Thanks Nils for the suggestion but I would like to be able to set different functions for _SampleFunc() not only the 'normal' function which I gave an example. Is there any way to make it more general?

Well, yes, but the function doing the wrapping needs to be a cdef function because you have to pass it a parameter that is not a python object (that's the point of wrapping it):

cdef object wrap_SampleFunc(FuncDef f):
    cdef _SampleFunc S = _SampleFunc()
    S.func = f
    return S
 

[Zahra Sheikh]

If I define the "normal" function as a c function as follows in the "ars.pyx" script:

cdef void normal(
                        int n,
                       double* u,
                       double* yu,
                       double* ypu
                        ):
      cdef int i
     
      for i in prange(n, nogil=True):
          yu[i]=-u[i]*u[i]*0.5                                                              
          ypu[i]=-u[i]                                                                   
      return                                                    
Is it possible to import this "c" function in my python code via
import ctypes
ars = ctypes.CDLL('./ars.so')
ars.normal.argtypes = (ctypes.c_int, ctypes.POINTER(ctypes.c_double), ctypes.POINTER(ctypes.c_double), ctypes.POINTER(ctypes.c_double))

and pass as argument to the python function?
Regards,
Z.


[Stefan Behnel]

Zahra Sheikh schrieb am 26.09.2017 um 16:22:
> Thanks Nils for the suggestion but I would like to be able to set different
> functions for _SampleFunc() not only the 'normal' function which I gave an
> example. Is there any way to make it more general?

You can set the "func" pointer to a different function. The idea is to
create multiple instances of the _SampleFunc type which point to different
C functions. Then the user can choose the right one and pass it into your
processing function, which then unpacks the function pointer and calls it.

Stefan

[Zahra Sheikh]

Sorry that I write here back and forth. I am trying to learn and make my code work. So based on what you have been replied I did something like that:

 %%cython
import cython
from cython.parallel import prange
ctypedef void (*func_t)(int, double *, double *, double *)
cdef class _SampleFunc:
     cdef void (*func)(int, double *, double *, double *)

       
cdef void normal(
                 int n,
                 double* u,
                 double* yu,
                 double* ypu
                ):         
      cdef int i         
      for i in prange(n, nogil=True):
          yu[i] =-u[i]*u[i]*0.5                                                              
          ypu[i]=-u[i]                                                                   
      return 
 

def wrapper(func_t f):
   S = _SampleFunc()
   S.func = &f
   return S

The error message that I got is

Error compiling Cython file:
------------------------------------------------------------
...

          yu[i] =-u[i]*u[i]*0.5                                                              
          ypu[i]=-u[i]                                                                   
      return   

def wrapper(func_t f):
   S = _SampleFunc()
   S.func = &f
           ^
------------------------------------------------------------

/home/.cache/ipython/cython/_cython_magic_ee4eb268e7fd035a.pyx:20:12: Cannot assign type 'func_t *' to 'void (*)(int, double *, double *, double *)'

So would anybody please clarify again how I also can make a python wrapper for the normal function?

Regards,
Z.

[Nils Bruin]

Have you tried it without the "&"? Since there already is a "*" in your type definition it looks like using "&" would give you a pointer to a pointer.

[Zahra Sheikh]

Without &, I will get this error :

Error compiling Cython file:
------------------------------------------------------------
...

     for i in prange(n, nogil=True):
         yu[i] =-u[i]*u[i]*0.5                                                             
         ypu[i]=-u[i]                                                                  
     return

def wrapper(func_t f):

           ^
------------------------------------------------------------

/home/.cache/ipython/cython/_cython_magic_b3b08b1f0eb1b96.pyx:19:12: Cannot convert Python object argument to type 'func_t'

[Nils Bruin]

On Wednesday, September 27, 2017 at 11:02:40 AM UTC-7, Zahra Sheikh wrote:

Without &, I will get this error :
Error compiling Cython file:
------------------------------------------------------------
...
     for i in prange(n, nogil=True):
         yu[i] =-u[i]*u[i]*0.5                                                             
         ypu[i]=-u[i]                                                                  
     return

def wrapper(func_t f):
           ^
------------------------------------------------------------

/home/.cache/ipython/cython/_cython_magic_b3b08b1f0eb1b96.pyx:19:12: Cannot convert Python object argument to type 'func_t'

That error exactly identifies the problem: a "def" function will get python objects as arguments and cython does not know how to convert a python object to a "func_t".
defining it "cdef object wrapper(func_t f)" shouldn't have that problem.

Remember that python functions can only receive and return python objects. Anything else needs to happen with cdef functions. For any func_t object, you're going to have to produce a wrapper (in cython or C code!) to be able to handle the object in python. Since compiling C-functions is a pretty static operation anyway, declaring the python wrapper statically with it isn't much of a loss. Only once you start dynamically loading libraries that provide functions of the right signature does producing wrappers become a dynamic necessity.

[Nils Bruin]

---------- Forwarded message ----------
From: Zahra Sheikh <sheikh...@gmail.com>
Date: Wed, Sep 27, 2017 at 5:59 AM
Subject: Private message regarding: [cython-users] Re: passing a pointer function to another function which can be called in python
To: bruin...@gmail.com

If I define the "normal" function in the "ars.pyx" script as follow:

cdef void normal(
                 int n,
                 double* u,
                 double* yu,
                 double* ypu
                ):

      #evaluate log  of normal distribution and its derivative
      cdef int i

     
      for i in prange(n, nogil=True):
          yu[i]=-u[i]*u[i]*0.5                                                              
          ypu[i]=-u[i]                                                                   
      return                                                    

Is it possible to import to import it via these lines in my python code:

import ctypes
ars = ctypes.CDLL('./ars.so')
ars.normal.argtypes = (ctypes.c_int, ctypes.POINTER(ctypes.c_double), ctypes.POINTER(ctypes.c_double), ctypes.POINTER(ctypes.c_double))

------------------

I don't know. You could try and see? Whether that works or not is quite independent of cython.

In that case, ars.normal would already be a wrapper of the function pointer, so in that case you'd need to write your function that is supposed to receive the function to accept a wrapper of that type and you'd have to figure out a way to access the pointer.

Note that, if you go this route, your type-safety is entirely based on your ars.normal.argtypes definition. You should probably declare the return type somewhere as well.

[Zahra Sheikh]

Sorry again for my posts here:

I ended up writing the following python function (wrapper) in order to get access to my "initial" and "sample" C-type functions:
def py_ars(int ns, int m, double emax,

           np.ndarray[ndim=1, dtype=np.float64_t] x,
           np.ndarray[ndim=1, dtype=np.float64_t] hx,

           np.ndarray[ndim=1, dtype=np.float64_t] hpx,
           int num,
           f #log of the distribution
           ):

    cdef np.ndarray[ndim=1, dtype=np.float64_t] rwv, sp
    cdef np.ndarray[ndim=1, dtype=np.int64_t] iwv
    # initializing arrays
    rwv = np.zeros(ns*6+15, dtype=np.float64)
    iwv = np.zeros(ns+7, dtype=np.int64)
    sp = np.zeros(num, dtype=np.float64)
   
    cdef double xlb = np.min(x)
    cdef double xub = np.max(x)
    cdef int lb=0
    cdef int ub=0
    cdef int ifault = 999
    cdef double beta = 0.

    initial(&ns, &m, &emax,
            &x[0], # passing array by reference
            &hx[0], # passing array by reference
            &hpx[0], # passing array by reference
            &lb, # transforming bool in int
            &xlb,
            &ub, # transforming bool in int
            &xub,
            &ifault, # passing integer variable by reference
            <int *>(&iwv[0]), # passing array by reference
            &rwv[0] # passing array by reference
            )
    FTYPE = ctypes.CFUNCTYPE(None, # return type
                             ctypes.POINTER(ctypes.c_double),
                             ctypes.POINTER(ctypes.c_double),
                             ctypes.POINTER(ctypes.c_double))
    f = FTYPE(f) # convert Python callable to ctypes function pointer

    # a rather nasty line to convert to a C function pointer
    cdef func_t f_ptr = (<func_t*><size_t>ctypes.addressof(f))[0]
    cdef int i
    for i from 0 <= i <num:
        sample(
               <int *>(&iwv[0]), # passing array by reference
               &rwv[0], # passing array by reference
               f_ptr,
               &beta, # passing double variable by reference
               &ifault, # passing integer variable by reference
               )
        sp[i] = beta

    return sp    

1) The original problem was that I needed to have a python callable function which wraps both these functions and I would be able to pass a pointer function (f) as an argument to this function.

2) Another problem was that the function that I would like to pass as argument must have pointers as its arguments (I would like to pass different functions --distributions-- as argument to the above function) and now the problem is how to write this function to be callable from my python script. That is how I did it

 def normal(double u,
           double yu,
           double ypu
           ):         
    yu=-u*u*0.5                                                              
    ypu=-u 
    return

def normal_ctypes( u, yu, ypu):
    u_as_ctypes_double   = (ctypes.c_double).from_address(ctypes.addressof(u))
    yu_as_ctypes_double  = (ctypes.c_double).from_address(ctypes.addressof(yu))
    ypu_as_ctypes_double = (ctypes.c_double).from_address(ctypes.addressof(ypu))
    normal(u_as_ctypes_double, yu_as_ctypes_double, ypu_as_ctypes_double)

3) The line of my code to call "py_ars" in my python script is as follows:

import numpy as np
from ars import py_ars, normal_ctypes
sp=py_ars(ns, m, emax, x, hx, hpx, num, normal_ctypes)

My ars.pyx gets compiled without any error, but I got this error message when I run my python script:
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 315, in 'calling callback function'
  File "ars.pyx", line 620, in ars.normal_ctypes
    normal(u_as_ctypes_double, yu_as_ctypes_double, ypu_as_ctypes_double)
  File "ars.pyx", line 608, in ars.normal
    def normal(double u,
TypeError: a float is required

So apparently I made a mistake defining the double type pointers as argument of "normal_ctypes" function. Any suggestion how it should be modified?

I attached my codes in case you want to check it out.

Thanks in advance.

Regards,
Z.

On Wednesday, 27 September 2017 19:58:05 UTC+2, Nils Bruin wrote:

[Nils Bruin]

On Thursday, September 28, 2017 at 4:50:43 AM UTC-7, Zahra Sheikh wrote:

2) Another problem was that the function that I would like to pass as argument must have pointers as its arguments (I would like to pass different functions --distributions-- as argument to the above function) and now the problem is how to write this function to be callable from my python script. That is how I did it

 def normal(double u,
           double yu,
           double ypu
           ):         
    yu=-u*u*0.5                                                              
    ypu=-u 
    return

This is now a PYTHON function that expects 3 "object" parameters that, upon entry, immediately get converted to C double arguments, using cython conversion heuristics. So the arguments should be python objects for which cython knows how to convert to C doubles.

This function suffers most of the inefficiencies of PYTHON functions when called.

 

def normal_ctypes( u, yu, ypu):
    u_as_ctypes_double   = (ctypes.c_double).from_address(ctypes.addressof(u))
    yu_as_ctypes_double  = (ctypes.c_double).from_address(ctypes.addressof(yu))
    ypu_as_ctypes_double = (ctypes.c_double).from_address(ctypes.addressof(ypu))
    normal(u_as_ctypes_double, yu_as_ctypes_double, ypu_as_ctypes_double)

This is a PYTHON function that takes as parameters 3 python objects. You then take each of these parameters and apply ctypes.addressof (which means that the parameters should be ctypes objects). You then take that address and construct a ctypes.c_double object out of it. So you've just constructed a a new wrapper around the c_double that you presumably already had (but have lost typesafety in the process because you didn't check that u really was a c_double.

 

You then pass these new wrappers to "normal". It would be equivalent (and a bit faster) to just call normal(u,yu,ypu) immediately.

Both fail at runtime, because the arguments (of type ctype.c_double) aren't python objects that cython knows how to turn into C doubles.

If you'd call normal(u.value,yuvalue,ypu.value) it would work, because then you'd be making a python float object with value equal to the ctypes.c_double object that you have.

----

Your original problem was that you had a c-function

T1 f(T2 a, T3 b, T4 c, T5 d)

which you wanted a wrapper for so that you can handle it in python and you wanted a way to unwrap the wrapper.

A was pointed out, writing a cython file that links against the library that provides f to implement and instantiate the wrapper (using wrap_SampleFunc) gives a way of doing that. That worked, right?

---

Your additional requirement was that you want to get your initial handle on the function using ctypes.CDLL, meaning that you have a ctypes._FuncPtr object

That wrapper works just as well, but you need a way to unwrap it. You could go and look at the implementation of _FuncPtr and write cython declarations for it that expose the required field.

Or you could use something along the lines of:

to get the relevant pointer as a python integer first and cast that to the requisite pointer type:

<_SampleFunc> <void *> ctypes.addressof(wrapped_function)

This will be a bit slower because it makes an intermediate python integer.

So perhaps you want something like:

def wrapper_from_ctypes_function(F):

    w = _SampleFunc()

    w = <func_t> ctypes.addressof(F)

    return w

to convert the one type of wrapper to the other.

Again, the fully cython approach has advantage that you get the c-compiler to do type checking for you. By mixing two ways of wrapping C objects (cython and ctypes) you are burdening yourself with having to convert between the two.

[Chris Barker - NOAA Federal]

On Sep 28, 2017, at 8:09 AM, Nils Bruin <bruin...@gmail.com> wrote:
...

Again, the fully cython approach has advantage that you get the c-compiler to do type checking for you. By mixing two ways of wrapping C objects (cython and ctypes) you are burdening yourself with having to convert between the two.

Which brings up the question:: Why are trying to mix ctypes and cython?

They are two quite different ways of calling c from python. The main difference being how much happens at compile time vs run time. Ctypes has the advantage of being able to write pure python that can call a pre-compiled lib. But if you are compiling a cpython module anyway, there is no advantage.

But it does complicate things !

-CHB

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Zahra Sheikh]

Using the comments above to make a python wrapper for the 'normal' function :
import ctypes

ctypedef void (*func_t)(double *, double *, double *)
cdef class _SampleFunc:
     cdef void (*func)(double *, double *, double *)

def wrapper_from_ctypes_function(F):
    w = _SampleFunc()
    w = <func_t> ctypes.addressof(F)
    return w

 raises couple of error messages:

Error compiling Cython file:
------------------------------------------------------------
...

cdef class _SampleFunc:
     cdef void (*func)(double *, double *, double *)

def wrapper_from_ctypes_function(F):
    w = _SampleFunc()
    w = <func_t> ctypes.addressof(F)

       ^
------------------------------------------------------------

ars.pyx:688:8: Casting temporary Python object to non-numeric non-Python type

Error compiling Cython file:
------------------------------------------------------------
...

cdef class _SampleFunc:
     cdef void (*func)(double *, double *, double *)

def wrapper_from_ctypes_function(F):
    w = _SampleFunc()
    w = <func_t> ctypes.addressof(F)

       ^
------------------------------------------------------------

ars.pyx:688:8: Python objects cannot be cast to pointers of primitive types

Error compiling Cython file:
------------------------------------------------------------
...

cdef class _SampleFunc:
     cdef void (*func)(double *, double *, double *)

def wrapper_from_ctypes_function(F):
    w = _SampleFunc()
    w = <func_t> ctypes.addressof(F)

       ^
------------------------------------------------------------

[Nils Bruin]

On Thursday, September 28, 2017 at 11:49:42 AM UTC-7, Zahra Sheikh wrote:

Using the comments above to make a python wrapper for the 'normal' function :
import ctypes

ctypedef void (*func_t)(double *, double *, double *)
cdef class _SampleFunc:
     cdef void (*func)(double *, double *, double *)

def wrapper_from_ctypes_function(F):
    w = _SampleFunc()
    w = <func_t> ctypes.addressof(F)
    return w

 raises couple of error messages:



Error compiling Cython file:
------------------------------------------------------------
...
cdef class _SampleFunc:
     cdef void (*func)(double *, double *, double *)

def wrapper_from_ctypes_function(F):
    w = _SampleFunc()
    w = <func_t> ctypes.addressof(F)
       ^
------------------------------------------------------------

ars.pyx:688:8: Casting temporary Python object to non-numeric non-Python type

Cython experts may have something to say about better workarounds. Clearly, you're breaking type safety by these casts and cython is trying to protect you from doing that. One way around it is to fake cython with an intermediate cast. Something like

 w = <func_t><void *><intptr_t> ctypes.addressof(F)

or some subset or variation of such casts.I don't understand the exact rules but have always been able to get the desired result with some experimentation (in less extreme cases than the one you're facing here though)

[Stefan Behnel]

Am 28. September 2017 20:49:42 MESZ schrieb Zahra Sheikh:
>Using the comments above to make a python wrapper for the 'normal'
>function
>:
>import ctypes
>
>ctypedef void (*func_t)(double *, double *, double *)
>cdef class _SampleFunc:
> cdef void (*func)(double *, double *, double *)

You already have a typedef for that function type. Just use it for this attribute as well

cdef func_t func

>def wrapper_from_ctypes_function(F):
> w = _SampleFunc()
> w = <func_t> ctypes.addressof(F)
> return w

Seriously, get rid of ctypes in your code. It complicates things and allows users to crash your library by simply passing anything inappropriate.

Look at the example that I gave you. It has all you need. If you want to export multiple functions, create multiple instances of _SampleFunc and assign different cdef functions to their func attribute, then let users choose one and pass it back to you.

Or are you intending to let users define their own functions?

Stefan

[Zahra Sheikh]

If you do not mind I recap again because it is getting really confusing for me to sum up with a solution and I guess it is not clear what I am trying to do:

I want to write a python wrapper function for a C-type function which gets a pointer function as its argument. I have different log-probability distributions that I would like to pass as argument to both this python wrapper and consequently to the C-type function that is embedded in the python wrapper.
My problems are:
1) how to pass different functions as argument to a python wrapper while they can also be callable from a python script?
for instance the following function:

cdef class _SampleFunc:
       cdef void (*func)(double *, double *, double *)

def normal(double* u,
                  double* yu,
                  double* ypu):         
     yu[0] = -u[0]*u[0]*0.5                                                              
     ypu[0]= -u[0]
     return
2) how can I make the above function (for instance normal function) be callable from my python script. I\d like to pass it as an argument to the python wrapper, it becomes unwrapped and then this function (e.g. normal function) gets passed as argument to a function inside the python wrapper (which is a c-type function e.g. sample function)?

 def py_ars(int ns, int m, double emax,
           np.ndarray[ndim=1, dtype=np.float64_t] x,
           np.ndarray[ndim=1, dtype=np.float64_t] hx,
           np.ndarray[ndim=1, dtype=np.float64_t] hpx,
           int num,

           _SampleFunc f #Different functions can pass as argument here
           ):

    cdef np.ndarray[ndim=1, dtype=np.float64_t] sp

    cdef double beta = 0. 

    cdef int i
   
    for i from 0 <= i <num:

       
        sample(      #a c-type  function which three pointer arrays, a pointer function and a double pointer pass as its arguments
                   &x[0]), # passing array by reference

                   &hx[0], # passing array by reference
                   &hpx[0],# passing array by reference

                   &f,

                   &beta, # passing double variable by reference

                   )
       
        sp[i] = beta

    return sp  

This above way of writing it is not working. How should it be done. Thanks again! 

[Stefan Behnel]

Hi,

sorry, but your code below is a complete mess. I'll just add comments as
far as I can decypher it.

Zahra Sheikh schrieb am 29.09.2017 um 14:17:
> If you do not mind I recap again because it is getting really confusing for
> me to sum up with a solution and I guess it is not clear what I am trying
> to do:
>
> I want to write a python wrapper function for a C-type function which gets
> a pointer function as its argument. I have different log-probability
> distributions that I would like to pass as argument to both this python
> wrapper and consequently to the C-type function that is embedded in the
> python wrapper.
> My problems are:
> 1) how to pass different functions as argument to a python wrapper while
> they can also be callable from a python script?

Create a Python wrapper object for them that you can pass through Python
code, pack and unpack that object at need, and call the C functions from
its pointer in Cython.

> for instance the following function:
>

> *cdef class _SampleFunc: cdef void (*func)(double *, double *, double
> *) *
>
>
>
>
>
>
> *def normal(double* u, double* yu,

> double* ypu): yu[0] =
> -u[0]*u[0]*0.5

> ypu[0]= -u[0] return*

That should be a cdef function. It takes pointers as arguments and is
therefore not callable from Python.

> 2) how can I make the above function (for instance normal function) be
> callable from my python script.

It cannot be called from Python because Python does not know pointers. You
can implement a separate def-function for it that accepts Python object
arguments, converts them in some way, and then passes them into the C function.

> I\d like to pass it as an argument to the
> python wrapper, it becomes unwrapped and then this function (e.g. normal
> function) gets passed as argument to a function inside the python wrapper
> (which is a c-type function e.g. sample function)?

Yes, that's the approach.


> * def py_ars(int ns, int m, double emax, np.ndarray[ndim=1,

> dtype=np.float64_t] x, np.ndarray[ndim=1, dtype=np.float64_t]
> hx, np.ndarray[ndim=1, dtype=np.float64_t] hpx, int

> num, _SampleFunc f *#Different functions can pass as argument here
> * ):*
>
>
>
>
>
>
> * cdef np.ndarray[ndim=1, dtype=np.float64_t] sp cdef double beta = 0.
> cdef int i for i from 0 <= i <num:*
>
> *sample( * #a c-type function which three pointer arrays, a

> pointer function and a double pointer pass as its arguments

> * &x[0]),* # passing array by reference
> * &hx[0],* # passing array by reference
> *&hpx[0],*# passing array by reference
> *&f,*

This is wrong. It gives you the pointer to the Python "_SampleFunc" object,
which is useless. Instead, unpack the C function (f.func) and pass that.

You should also declare the argument in the "py_ars" signature as
"_SampleFunc f not None", because Cython will otherwise allow users to pass
None.


> * &beta*, # passing double variable by reference
> *)*
>
> *sp[i] = beta*
>
> *return sp*

Stefan

[Zahra Sheikh]

Sorry for the errors in normal function. I wrote this python wrapper for normal function


cdef void normal(double* u,

                 double* yu,
                 double* ypu):         
     yu[0] = -u[0]*u[0]*0.5                                                              
     ypu[0]= -u[0]

     print yu[0],ypu[0]
     return
 

def normal_distr(double u,
                 double yu,
                 double ypu):
    cdef double[::1] x=np.asarray([u])
    cdef double[::1] hx=np.asarray([yu])
    cdef double[::1] hpx=np.asarray([ypu])
    normal(&x[0],&hx[0],&hpx[0])

Is there a better way to write it?

[Zahra Sheikh]

 I wrote this python wrapper for the normal function

cdef void C_normal(double* u,

                 double* yu,
                 double* ypu):         
     yu[0] = -u[0]*u[0]*0.5                                                              
     ypu[0]= -u[0]     

     return

 cdef class _SampleFunc:
       cdef void (*func)(double *, double *, double *)

def normal_wrapper(double x,double hx, double hpx):
    cdef _SampleFunc S=_SampleFunc()
    S.func = &C_normal(&x,&hx,&hpx)
    return S

based on again what has been suggested here and now I am getting this error message:

    S.func = &C_normal(&x,&hx,&hpx)
            ^
------------------------------------------------------------

ars.pyx:618:13: Taking address of non-lvalue (type void)

How should I fix this error message?

Thanks a lot!
Z.

[Nils Bruin]

You can fix the error by not trying to take the address of a "void" object (the (lack of) return value of C_normal(&x,&hx,&hpx))

However, I don't see why "normal_wrapper" should take "double"s as arguments.

Your code suggests to me that you're confused about what you want to accomplish. You might want to take a step back (perhaps make a prototype in python first?) and get your design decisions and the way you want to realize them clear to yourself. Your earlier attempts looked closer to working code.
 

[Zahra Sheikh]

Well I am getting really confused because I thought I am following the suggestions that has been given here, in order to first write a python wrapper for the normal function which also can be passed to the python wrapper function (py_ars) as argument but it seems I misunderstood the comments. I will appreciate if someone explain what I should do again?

 Again This is the piece of code:

cdef void C_normal(double* u,
                 double* yu,
                 double* ypu):         
     yu[0] = -u[0]*u[0]*0.5                                                              
     ypu[0]= -u[0]     
     return
 

def normal_wrapper(double x,double hx, double hpx):
    cdef _SampleFunc S=_SampleFunc()

    S.func = C_normal(&x,&hx,&hpx)
    return S

                
cdef class _SampleFunc:
       cdef void (*func)(double *, double *, double *)

def py_ars(int ns, int m, double emax,
           np.ndarray[ndim=1, dtype=np.float64_t] x,
           np.ndarray[ndim=1, dtype=np.float64_t] hx,
           np.ndarray[ndim=1, dtype=np.float64_t] hpx,
           int num,

           _SampleFunc f not None
           ):
    cdef np.ndarray[ndim=1, dtype=np.float64_t]  rwv, sp

    cdef np.ndarray[ndim=1, dtype=np.int64_t] iwv

    rwv = np.zeros(ns*6+15, dtype=np.float64)
    iwv = np.zeros(ns+7, dtype=np.int64)
    sp = np.zeros(num, dtype=np.float64)

    cdef int ifault = 0
    cdef double beta = 0

    cdef int i
   
    for i from 0 <= i <num:
       

        sample(
               <int *>(&iwv[0]), # passing array by reference
               &rwv[0], # passing array by reference
               f.func,
               &beta, # passing double variable by reference
               &ifault, # passing integer variable by reference

               )
       
        sp[i] = beta

    return sp  

Thanks in advance.

[Nils Bruin]

On Monday, October 2, 2017 at 11:25:07 AM UTC+2, Zahra Sheikh wrote:

Well I am getting really confused because I thought I am following the suggestions that has been given here, in order to first write a python wrapper for the normal function which also can be passed to the python wrapper function (py_ars) as argument but it seems I misunderstood the comments. I will appreciate if someone explain what I should do again?

OK, just to check that our instructions actually were for something that's possible, with the follow cython file:

----------------------------------------------------

ctypedef double (* func_t)(double)
cdef class wrapper:
    cdef func_t wrapped
    def __call__(self, value):
        return self.wrapped(value)
    def __unsafe_set(self, ptr):
        self.wrapped = <func_t><void *><size_t>ptr
def list_apply(wrapper w, L):
    for i in range(len(L)):
        L[i] = w.wrapped(L[i])
cdef double S(double x):
    return x*x
cdef double D(double x):
    return x+x
cdef wrapper make_wrapper(func_t f):
    cdef wrapper W=wrapper()
    W.wrapped=f
    return W
wrapS=make_wrapper(S)
wrapD=make_wrapper(D)

---------------------------------------------------------

you can do things like

sage: load("load.pyx")
Compiling ./load.pyx...
sage: L=[1,2,3,4,5]
sage: list_apply(wrapS,L)
sage: L
[1.0, 4.0, 9.0, 16.0, 25.0]
sage: list_apply(wrapD,L)
sage: L
[2.0, 8.0, 18.0, 32.0, 50.0]

(adjust appropriately to use directly in python instead of in sage)

I thought that if you'd have something like "libS.so" containing the result of the C code:

double S(double x) {
    return x*x;
}

it should be possible to do something along the lines of

sage: import ctypes
sage: libS=ctypes.CDLL("libS.so")
sage: W=wrapper()
sage: W.__unsafe_set(ctypes.addressof(libS.S))
sage: list_apply(W,L)

but that segfaults for me, so it seems that ctypes.addressof is not a good way of retrieving the pointer value from a ctypes._FuncPtr object.

The code above is only for illustrative purposes. It has no performance benefit at all. Yes, list_apply calls the function contained in the wrapper directly, without python function call overhead, but before doing that if goes through the process of converting the argument (given as a python object) to a C_double, and it does the same in reverse for the result. But once you've mastered the mechanics of passing around function pointers in wrappers, you can figure out how to get performance benefit from it.

If you absolutely want to use ctypes, you should probably seek information from experts on ctypes on how to get the actual pointer value out of a _FuncPtr object.