Passing objects supporting the buffer protocol to pointer parameters

[Graham Markall]

Hi,

Motivated by an enquiry by a Numba user (
https://github.com/numba/numba/issues/1443 ), I found that it's not
presently possible to pass an object supporting the buffer protocol
directly to a pointer parameter in CFFI, though I did find this mention
that eventually support for this may be added:

https://groups.google.com/forum/#!msg/python-cffi/9uc6aqX7z0Q/Q8cb8BVtt-IJ

I tried experimenting a little with this to see if I could make steps
towards its implementation, using the following code as a test:

from numba import cffi_support, jit
import numpy as np
import cffi

# Create the vectormaths module

defs = "void vsSin(int n, float* x, float* y);"
source = """\
void vsSin(int n, float* x, float* y) {
    int i;
    for (i=0; i<n; i++)
        y[i] = sin(x[i]);
}"""


ffi = cffi.FFI()
ffi.set_source('vectormaths', source)
ffi.cdef(defs, override=True)
ffi.compile()


# Import the compiled module

import vectormaths
cffi_support.register_module(vectormaths)
vsSin = vectormaths.lib.vsSin


# Pass an object supporting the buffer protocol to a float*
# Not currently supported

def pass_buffer_to_ptr(x):
    y = np.empty_like(x)
    vsSin(len(x), x, y)
    return y

x = np.arange(10).astype(np.float32)
pass_buf_y = pass_buffer_to_ptr(x)


# Use ffi.from_buffer. Currently supported

def from_buffer_to_ptr(x):
    y = np.empty_like(x)
    vsSin(len(x), ffi.from_buffer(x), ffi.from_buffer(y))
    return y

x = np.arange(10).astype(np.float32)
from_buf_y = from_buffer_to_ptr(x)


# These two versions do produce the same thing
assert np.all(pass_buf_y == from_buf_y)

print(pass_buf_y)
print(from_buf_y)

Presently calling the pass_buffer_to_ptr function fails, but I was able
to get it to work on CPython with a small modification to the
_prepare_pointer_call_argument function:
https://bitbucket.org/gmarkall/cffi/commits/8aa7cd36813529838d4001c666650dea38447117

Aside from some validation to check the format of the buffer is suitable
for the type of the parameter being passed to, I see also a couple of
other difficulties with proceeding:

* The Py_buffer view that is now created gets leaked, as I can't see a
clean way to "remember" it through the interface of
_prepare_pointer_call_argument so that it can be cleaned up after the
call to the actual C function. One possibility I could see would be to
return a particular sentinel value (e.g. -1 for general failure, -2 for
"failure, but supports buffer protocol", as opposed to the present
situation where any negative number seems to indicate failure) so that
the code to create the Py_buffer can be inside the calling function (in
my example, _cffi_f_vsSin), and therefore it can also clean up more
easily after the call to the C function. Would this be a suitable way to
go?

* I don't have any experience with PyPy, so I'd expect to struggle with
making any progress on support for this feature with PyPy.


Any guidance on how to proceed in terms of the right way to
structure/implement changes to make this into an acceptable contribution
to cffi would be appreciated. Alternately, if pursuing this path is
unlikely to result in a worthwhile contribution, please let me know :-)


Best regards,
Graham Markall.

[Armin Rigo]

Hi Graham,

The function ffi.from_buffer() is more recent than the mail you point
to. So from_buffer() was added a some point and this was supposed to
close that old issue. By now, I'd say it's not worth it to *also*
support direct calls without from_buffer(). It creates various issues
that are so far nicely contained in the from_buffer() function.

You've found the one about the time during which the Py_buffer needs
to be kept alive. There are more similar issues in other cases, like
if you use the ABI mode.

But the main issue I have is that for the user, direct support might
give results that are a bit strange: say, if you have an
array.array('i'), then you could pass directly to a "long *" argument
and not get any error or warning. You would expect that the type of
the array is matched against the type of the argument, as CFFI (and C)
generally does; but this is not possible in general from a Python
buffer object. (I know that numpy provides some "typed" buffer view,
but that's the exception rather than the rule for Python buffer
objects. Even CFFI itself does not try to type the result of
ffi.buffer(ffi.new("int[]", 10)).) With the explicit
ffi.from_buffer(), you make it clear that you're casting to a generic
"char *", and then passing this "char *" to a "long *" argument; it is
then more natural that you don't get any typing error.


A bientôt,

Armin.

[Graham Markall]

Hi Armin,

Many thanks for the quick response - I hadn't realised that from_buffer was actually the solution! :-)

I think I will try to support from_buffer in Numba in order to work around my original problem (which should not be too difficult).

Best regards,

Graham.

--
-- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
---
You received this message because you are subscribed to the Google Groups "python-cffi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-cffi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.