Writing a C callback function for use with ndimage.generic_filter

7 views
Skip to first unread message

Juan Nunez-Iglesias

unread,
Oct 27, 2016, 11:20:52 PM10/27/16
to numba...@continuum.io
I stumbled across the cfunc Numba decorator this morning, with much excitement. But I'm having trouble using it with ndimage.generic_filter:

=============================================

In [3]: def mean(arr):
   ...:     i = 0.
   ...:     tot = 0.
   ...:     for val in arr:
   ...:         tot += val
   ...:         i += 1.
   ...:     return tot / i
   ...: 

In [4]: cmean = cfunc("float64(float64[:])")(mean)

In [5]: from skimage import data

In [6]: coins = data.coins() / 255

In [8]: from scipy import ndimage as ndi

In [9]: fp = ndi.generate_binary_structure(2, 2)

In [14]: coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-ccdc82fc8ce6> in <module>()
----> 1 coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp)

/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/numba/utils.py in __get__(self, instance, type)
    276         if instance is None:
    277             return self
--> 278         res = instance.__dict__[self.name] = self.func(instance)
    279         return res
    280 

/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/numba/ccallback.py in ctypes(self)
    156         A ctypes function object representing the C callback.
    157         """
--> 158         ctypes_args = [to_ctypes(ty) for ty in self._sig.args]
    159         ctypes_restype = to_ctypes(self._sig.return_type)
    160         functype = ctypes.CFUNCTYPE(ctypes_restype, *ctypes_args)

/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/numba/ccallback.py in <listcomp>(.0)
    156         A ctypes function object representing the C callback.
    157         """
--> 158         ctypes_args = [to_ctypes(ty) for ty in self._sig.args]
    159         ctypes_restype = to_ctypes(self._sig.return_type)
    160         functype = ctypes.CFUNCTYPE(ctypes_restype, *ctypes_args)

/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/numba/typing/ctypes_utils.py in to_ctypes(ty)
     80     if ctypeobj is None:
     81         raise TypeError("Cannot convert Numba type '%s' to ctypes type"
---> 82                         % (ty,))
     83     return ctypeobj
     84 

TypeError: Cannot convert Numba type 'array(float64, 1d, A)' to ctypes type

=============================================

I also tried an alternate formulation, naively hoping that Numba would gladly convert between array buffers and (double*, size) pairs, with no luck:

=============================================

In [21]: signature = types.float64(types.CPointer(types.float64), types.intc)

In [23]: def mean(arr, size):
    ...:     i = 0.
    ...:     tot = 0.
    ...:     arr = carray(arr, size)
    ...:     for idx in range(size):
    ...:         tot += arr[idx]
    ...:         i += 1.
    ...:     return tot / i
    ...:     
    ...: 

In [24]: cmean = cfunc(signature)(mean)

In [25]: coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-25-ccdc82fc8ce6> in <module>()
----> 1 coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp)

/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/scipy/ndimage/filters.py in generic_filter(input, function, size, footprint, output, mode, cval, origin, extra_arguments, extra_keywords)
   1243     mode = _ni_support._extend_mode_to_code(mode)
   1244     _nd_image.generic_filter(input, function, footprint, output, mode,
-> 1245                          cval, origins, extra_arguments, extra_keywords)
   1246     return return_value

TypeError: this function takes at least 2 arguments (1 given)

In [26]: coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp, extra_arguments=(np.sum(fp),))
---------------------------------------------------------------------------
ArgumentError                             Traceback (most recent call last)
<ipython-input-26-f40d45b26623> in <module>()
----> 1 coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp, extra_arguments=(np.sum(fp),))

/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/scipy/ndimage/filters.py in generic_filter(input, function, size, footprint, output, mode, cval, origin, extra_arguments, extra_keywords)
   1243     mode = _ni_support._extend_mode_to_code(mode)
   1244     _nd_image.generic_filter(input, function, footprint, output, mode,
-> 1245                          cval, origins, extra_arguments, extra_keywords)
   1246     return return_value

ArgumentError: argument 1: <class 'TypeError'>: expected LP_c_double instance instead of numpy.ndarray

=============================================

The relevant doc pages are Numba's cfunc page:
And SciPy's C callable instructions for ndimage:

Unfortunately the above is how far my interpretation of those pages have taken me. Can anyone on this list help me with this? I'll be happy to contribute the solution back to the docs if it's considered useful!

Thanks,

Juan.

Antoine Pitrou

unread,
Oct 28, 2016, 5:20:51 AM10/28/16
to numba...@continuum.io

Hello Juan,

On Fri, 28 Oct 2016 14:20:30 +1100
Juan Nunez-Iglesias <jni....@gmail.com>
wrote:
> I stumbled across the cfunc Numba decorator this morning, with much
> excitement. But I'm having trouble using it with ndimage.generic_filter:

The Scipy docs (*) don't mention that the function can be a ctypes
callable, so you'll only be able to pass a regular @jit function to it.

(*)
https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.ndimage.generic_filter.html

Regards

Antoine.


Travis Oliphant

unread,
Oct 28, 2016, 6:28:25 AM10/28/16
to Numba Users
I suspect the Numba project would welcome a PR that provides a PyCObject from a cfunc result, however.   Given the prevalence of these in SciPy, this would be a very useful way to wrap cfuncs.   You may actually be able to use ctypes to create one from the cfunc object. 

Best,

-Travis




--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users+unsubscribe@continuum.io.
To post to this group, send email to numba...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/20161028111943.25facd42%40fsol.



--

Travis Oliphant, PhD
Co-founder and CEO


@teoliphant

Juan Nunez-Iglesias

unread,
Oct 28, 2016, 7:42:47 AM10/28/16
to numba...@continuum.io
If anyone listening has good ideas about how to make this happen, I would certainly appreciate it! I must admit that Python/C interaction and compilers are far beyond my expertise, so I’m going to hope someone else picks up this particular baton!

Thanks so much nevertheless for a great project!

Juan.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users...@continuum.io.

To post to this group, send email to numba...@continuum.io.

Antoine Pitrou

unread,
Oct 28, 2016, 7:51:39 AM10/28/16
to numba...@continuum.io

Hi Travis,

On Fri, 28 Oct 2016 05:28:03 -0500
Travis Oliphant <tra...@continuum.io> wrote:
> I suspect the Numba project would welcome a PR that provides a PyCObject
> from a cfunc result, however. Given the prevalence of these in SciPy,
> this would be a very useful way to wrap cfuncs.

I'm not sure what the point would be. The purpose of using a cfunc is
to eliminate the Python function call overhead (parsing and unboxing
arguments). If you wrap a cfunc in a PyCObject, you re-add that
overhead and the result wouldn't be much different from a jit function,
performance-wise.

Regards

Antoine.


Travis Oliphant

unread,
Oct 28, 2016, 8:05:40 AM10/28/16
to Numba Users
Definitely you would not want to wrap the cfunc itself with the PyCObject.   The useful thing would be to wrap the "unboxed" function pointer the cfunc is pointing to with a PyCObject so that the code in ndimage "unboxes" it and you get the raw function pointer.      

I think a method on a cfunc that returns a PyCObject that is wrapping the underlying C-function pointer would be exactly what this use-case is looking for. 

-Travis



Regards

Antoine.


--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users+unsubscribe@continuum.io.
To post to this group, send email to numba...@continuum.io.

Antoine Pitrou

unread,
Oct 28, 2016, 8:13:06 AM10/28/16
to numba...@continuum.io
On Fri, 28 Oct 2016 07:05:18 -0500
Travis Oliphant <tra...@continuum.io> wrote:
> Definitely you would not want to wrap the cfunc itself with the PyCObject.
> The useful thing would be to wrap the "unboxed" function pointer the
> cfunc is pointing to with a PyCObject so that the code in ndimage "unboxes"
> it and you get the raw function pointer.

That presumes that ndimage.generic_filter() is able to do so :-) Perhaps
the current documentation isn't complete, but it doesn't seem to say
anything about that, or perhaps I'm looking in the wrong place?

> I think a method on a cfunc that returns a PyCObject that is wrapping the
> underlying C-function pointer would be exactly what this use-case is
> looking for.

If it's only the function pointer (without any signature information),
it can already be retrieved it as a Python integer using the `address`
attribute. I'm not sure anything more is needed:
http://numba.pydata.org/numba-doc/latest/reference/jit-compilation.html#c-callbacks

Regards

Antoine.


Travis Oliphant

unread,
Oct 28, 2016, 3:23:39 PM10/28/16
to Numba Users
This is the paragraph I was reading: 

"A few functions in the scipy.ndimage take a call-back argument. This can be a python function, but also a PyCObject containing a pointer to a C function. To use this feature, you must write your own C extension that defines the function, and define a Python function that returns a PyCObject containing a pointer to this function."


It seems that if you pass in to scipy.ndimage a PyCObject it will "unbox" the function-pointer --- getting one of these from the cfunc would enable Numba to be used to provide call-backs to scipy.ndimage (and the same pattern should work for other call-back functions as well). 

-Travis




Regards

Antoine.


--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users+unsubscribe@continuum.io.
To post to this group, send email to numba...@continuum.io.

Antoine Pitrou

unread,
Oct 28, 2016, 3:50:14 PM10/28/16
to numba...@continuum.io
On Fri, 28 Oct 2016 14:23:17 -0500
Travis Oliphant <tra...@continuum.io> wrote:
> This is the paragraph I was reading:
>
> "A few functions in the scipy.ndimage
> <https://docs.scipy.org/doc/scipy-0.18.1/reference/ndimage.html#module-scipy.ndimage>
> take
> a call-back argument. This can be a python function, but also a
> PyCObject containing
> a pointer to a C function. To use this feature, you must write your own C
> extension that defines the function, and define a Python function that
> returns a PyCObject containing a pointer to this function."

Thanks for the reference. I didn't know about that. It's a pity the
approach is entirely different from what was done in scipy.integrate :-)
https://docs.scipy.org/doc/scipy-0.18.1/reference/tutorial/integrate.html#faster-integration-using-ctypes

The documentation seems a bit vague and outdated (it mentions
PyCObject, which is gone in Python 3, though the Scipy source code
shows PyCapsule objects are accepted as well; also, the precise C
signatures accepted by the various ndimage functions is not given,
which will make it a bit tedious for users to exploit this possibility).

> From here:
> https://docs.scipy.org/doc/scipy-0.18.1/reference/tutorial/ndimage.html#extending-ndimage-in-c
>
> It seems that if you pass in to scipy.ndimage a PyCObject it will "unbox"
> the function-pointer --- getting one of these from the cfunc would enable
> Numba to be used to provide call-backs to scipy.ndimage (and the same
> pattern should work for other call-back functions as well).

Yes, this should be doable. The author of the cfunc, of course, must
ensure the proper C signature. Unlike the ctypes-based solution used
by scipy.integrate, this one doesn't provide any type safety.

Regards

Antoine.


nov...@gmail.com

unread,
Nov 1, 2016, 2:35:39 AM11/1/16
to Numba Public Discussion - Public, soli...@pitrou.net
It seems if PR #6509 is merged then numba need only support the LowLevelCallable interface:



-Dave
Reply all
Reply to author
Forward
0 new messages