Writing a C callback function for use with ndimage.generic_filter
3 views
Skip to first unread message
Juan Nunez-Iglesias
Oct 27, 2016, 11:20:52 PM10/27/16
to numba...@continuum.io
I stumbled across the cfunc Numba decorator this morning, with much excitement. But I'm having trouble using it with ndimage.generic_filter:
=============================================
In [3]: def mean(arr):
...: i = 0.
...: tot = 0.
...: for val in arr:
...: tot += val
...: i += 1.
...: return tot / i
...:
In [4]: cmean = cfunc("float64(float64[:])")(mean)
In [5]: from skimage import data
In [6]: coins = data.coins() / 255
In [8]: from scipy import ndimage as ndi
In [9]: fp = ndi.generate_binary_structure(2, 2)
In [14]: coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-14-ccdc82fc8ce6> in <module>()
----> 1 coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp)
/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/numba/utils.py in __get__(self, instance, type)
276 if instance is None:
277 return self
--> 278 res = instance.__dict__[self.name] = self.func(instance)
279 return res
280
/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/numba/ccallback.py in ctypes(self)
156 A ctypes function object representing the C callback.
157 """
--> 158 ctypes_args = [to_ctypes(ty) for ty in self._sig.args]
159 ctypes_restype = to_ctypes(self._sig.return_type)
160 functype = ctypes.CFUNCTYPE(ctypes_restype, *ctypes_args)
/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/numba/ccallback.py in <listcomp>(.0)
156 A ctypes function object representing the C callback.
157 """
--> 158 ctypes_args = [to_ctypes(ty) for ty in self._sig.args]
159 ctypes_restype = to_ctypes(self._sig.return_type)
160 functype = ctypes.CFUNCTYPE(ctypes_restype, *ctypes_args)
/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/numba/typing/ctypes_utils.py in to_ctypes(ty)
80 if ctypeobj is None:
81 raise TypeError("Cannot convert Numba type '%s' to ctypes type"
---> 82 % (ty,))
83 return ctypeobj
84
TypeError: Cannot convert Numba type 'array(float64, 1d, A)' to ctypes type
=============================================
I also tried an alternate formulation, naively hoping that Numba would gladly convert between array buffers and (double*, size) pairs, with no luck:
=============================================
In [21]: signature = types.float64(types.CPointer(types.float64), types.intc)
In [23]: def mean(arr, size):
...: i = 0.
...: tot = 0.
...: arr = carray(arr, size)
...: for idx in range(size):
...: tot += arr[idx]
...: i += 1.
...: return tot / i
...:
...:
In [24]: cmean = cfunc(signature)(mean)
In [25]: coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-25-ccdc82fc8ce6> in <module>()
----> 1 coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp)
/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/scipy/ndimage/filters.py in generic_filter(input, function, size, footprint, output, mode, cval, origin, extra_arguments, extra_keywords)
1243 mode = _ni_support._extend_mode_to_code(mode)
1244 _nd_image.generic_filter(input, function, footprint, output, mode,
-> 1245 cval, origins, extra_arguments, extra_keywords)
1246 return return_value
TypeError: this function takes at least 2 arguments (1 given)
In [26]: coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp, extra_arguments=(np.sum(fp),))
---------------------------------------------------------------------------
ArgumentError Traceback (most recent call last)
<ipython-input-26-f40d45b26623> in <module>()
----> 1 coins_mean = ndi.generic_filter(coins, cmean.ctypes, footprint=fp, extra_arguments=(np.sum(fp),))
/Users/nuneziglesiasj/anaconda/envs/ana3/lib/python3.5/site-packages/scipy/ndimage/filters.py in generic_filter(input, function, size, footprint, output, mode, cval, origin, extra_arguments, extra_keywords)
1243 mode = _ni_support._extend_mode_to_code(mode)
1244 _nd_image.generic_filter(input, function, footprint, output, mode,
-> 1245 cval, origins, extra_arguments, extra_keywords)
1246 return return_value
ArgumentError: argument 1: <class 'TypeError'>: expected LP_c_double instance instead of numpy.ndarray
=============================================
The relevant doc pages are Numba's cfunc page:
And SciPy's C callable instructions for ndimage:
Unfortunately the above is how far my interpretation of those pages have taken me. Can anyone on this list help me with this? I'll be happy to contribute the solution back to the docs if it's considered useful!
Thanks,
Juan.
Antoine Pitrou
Oct 28, 2016, 5:20:51 AM10/28/16
to numba...@continuum.io
Hello Juan,
On Fri, 28 Oct 2016 14:20:30 +1100
Juan Nunez-Iglesias <jni....@gmail.com>
wrote:
> I stumbled across the cfunc Numba decorator this morning, with much
> excitement. But I'm having trouble using it with ndimage.generic_filter:
The Scipy docs (*) don't mention that the function can be a ctypes
> excitement. But I'm having trouble using it with ndimage.generic_filter:
callable, so you'll only be able to pass a regular @jit function to it.
(*)
https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.ndimage.generic_filter.html
Regards
Antoine.
Travis Oliphant
Oct 28, 2016, 6:28:25 AM10/28/16
to Numba Users
I suspect the Numba project would welcome a PR that provides a PyCObject from a cfunc result, however. Given the prevalence of these in SciPy, this would be a very useful way to wrap cfuncs. You may actually be able to use ctypes to create one from the cfunc object.
Best,
-Travis
--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users+unsubscribe@continuum.io.
To post to this group, send email to numba...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/20161028111943.25facd42%40fsol.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.
Juan Nunez-Iglesias
Oct 28, 2016, 7:42:47 AM10/28/16
to numba...@continuum.io
If anyone listening has good ideas about how to make this happen, I would certainly appreciate it! I must admit that Python/C interaction and compilers are far beyond my expertise, so I’m going to hope someone else picks up this particular baton!
Thanks so much nevertheless for a great project!
Juan.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/CAMcnTE5Ry163-9ox%3D8pzz-4ddC-fs5DJ4Hn%2BMzcGDU%2Bq19ySnA%40mail.gmail.com.
Antoine Pitrou
Oct 28, 2016, 7:51:39 AM10/28/16
to numba...@continuum.io
Hi Travis,
On Fri, 28 Oct 2016 05:28:03 -0500
Travis Oliphant <tra...@continuum.io> wrote:
> I suspect the Numba project would welcome a PR that provides a PyCObject
> from a cfunc result, however. Given the prevalence of these in SciPy,
> this would be a very useful way to wrap cfuncs.
to eliminate the Python function call overhead (parsing and unboxing
arguments). If you wrap a cfunc in a PyCObject, you re-add that
overhead and the result wouldn't be much different from a jit function,
performance-wise.
Regards
Antoine.
Travis Oliphant
Oct 28, 2016, 8:05:40 AM10/28/16
to Numba Users
Definitely you would not want to wrap the cfunc itself with the PyCObject. The useful thing would be to wrap the "unboxed" function pointer the cfunc is pointing to with a PyCObject so that the code in ndimage "unboxes" it and you get the raw function pointer.
I think a method on a cfunc that returns a PyCObject that is wrapping the underlying C-function pointer would be exactly what this use-case is looking for.
-Travis
Regards
Antoine.
--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users+unsubscribe@continuum.io.
To post to this group, send email to numba...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/20161028135103.201352a9%40fsol.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.
Antoine Pitrou
Oct 28, 2016, 8:13:06 AM10/28/16
to numba...@continuum.io
On Fri, 28 Oct 2016 07:05:18 -0500
Travis Oliphant <tra...@continuum.io> wrote:
> Definitely you would not want to wrap the cfunc itself with the PyCObject.
> The useful thing would be to wrap the "unboxed" function pointer the
> cfunc is pointing to with a PyCObject so that the code in ndimage "unboxes"
> it and you get the raw function pointer.
That presumes that ndimage.generic_filter() is able to do so :-) Perhaps
Travis Oliphant <tra...@continuum.io> wrote:
> Definitely you would not want to wrap the cfunc itself with the PyCObject.
> The useful thing would be to wrap the "unboxed" function pointer the
> cfunc is pointing to with a PyCObject so that the code in ndimage "unboxes"
> it and you get the raw function pointer.
the current documentation isn't complete, but it doesn't seem to say
anything about that, or perhaps I'm looking in the wrong place?
> I think a method on a cfunc that returns a PyCObject that is wrapping the
> underlying C-function pointer would be exactly what this use-case is
> looking for.
it can already be retrieved it as a Python integer using the `address`
attribute. I'm not sure anything more is needed:
http://numba.pydata.org/numba-doc/latest/reference/jit-compilation.html#c-callbacks
Regards
Antoine.
Travis Oliphant
Oct 28, 2016, 3:23:39 PM10/28/16
to Numba Users
This is the paragraph I was reading:
"A few functions in the scipy.ndimage take a call-back argument. This can be a python function, but also a PyCObject containing a pointer to a C function. To use this feature, you must write your own C extension that defines the function, and define a Python function that returns a PyCObject containing a pointer to this function."
It seems that if you pass in to scipy.ndimage a PyCObject it will "unbox" the function-pointer --- getting one of these from the cfunc would enable Numba to be used to provide call-backs to scipy.ndimage (and the same pattern should work for other call-back functions as well).
-Travis
Regards
Antoine.
--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users+unsubscribe@continuum.io.
To post to this group, send email to numba...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/20161028141210.0b8b8cc0%40fsol.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.
Antoine Pitrou
Oct 28, 2016, 3:50:14 PM10/28/16
to numba...@continuum.io
On Fri, 28 Oct 2016 14:23:17 -0500
Travis Oliphant <tra...@continuum.io> wrote:
> This is the paragraph I was reading:
>
> "A few functions in the scipy.ndimage
> <https://docs.scipy.org/doc/scipy-0.18.1/reference/ndimage.html#module-scipy.ndimage>
Travis Oliphant <tra...@continuum.io> wrote:
> This is the paragraph I was reading:
>
> "A few functions in the scipy.ndimage
> take
> a call-back argument. This can be a python function, but also a
> PyCObject containing
> a pointer to a C function. To use this feature, you must write your own C
> extension that defines the function, and define a Python function that
> returns a PyCObject containing a pointer to this function."
Thanks for the reference. I didn't know about that. It's a pity the
> a call-back argument. This can be a python function, but also a
> PyCObject containing
> a pointer to a C function. To use this feature, you must write your own C
> extension that defines the function, and define a Python function that
> returns a PyCObject containing a pointer to this function."
approach is entirely different from what was done in scipy.integrate :-)
https://docs.scipy.org/doc/scipy-0.18.1/reference/tutorial/integrate.html#faster-integration-using-ctypes
The documentation seems a bit vague and outdated (it mentions
PyCObject, which is gone in Python 3, though the Scipy source code
shows PyCapsule objects are accepted as well; also, the precise C
signatures accepted by the various ndimage functions is not given,
which will make it a bit tedious for users to exploit this possibility).
> From here:
> https://docs.scipy.org/doc/scipy-0.18.1/reference/tutorial/ndimage.html#extending-ndimage-in-c
>
> It seems that if you pass in to scipy.ndimage a PyCObject it will "unbox"
> the function-pointer --- getting one of these from the cfunc would enable
> Numba to be used to provide call-backs to scipy.ndimage (and the same
> pattern should work for other call-back functions as well).
ensure the proper C signature. Unlike the ctypes-based solution used
by scipy.integrate, this one doesn't provide any type safety.
Regards
Antoine.
nov...@gmail.com
Nov 1, 2016, 2:35:39 AM11/1/16
to Numba Public Discussion - Public, soli...@pitrou.net
It seems if PR #6509 is merged then numba need only support the LowLevelCallable interface:
-Dave
Reply all
Reply to author
Forward
0 new messages