Buffer protocol and column-major (FORTRAN) arrays.

[Laurent Gautier]

Hi,

First of all thanks for cffi: I have started using it and it is quite a nice option to call C code from Python.

I have question about Python's buffer protocol and column-major arrays. I am understanding that the only way to create an object implementing the buffer protocol with cffi is to:

1. Create a buffer (using cffi's function `buffer`)
2. Create a memory view from the buffer
   
3. Call the memoryview's method `cast` specifying the type of the elements in the array

The result is a row-major array and there does not seem to exist a way to create a column major array (while Python's buffer protocol explicitly allows it).

Did I miss something ?

Best,

Laurent

[Armin Rigo]

Hi Laurent,

On 22 September 2018 at 21:14, Laurent Gautier <lgau...@gmail.com> wrote:
> Create a buffer (using cffi's function `buffer`)
> Create a memory view from the buffer
> Call the memoryview's method `cast` specifying the type of the elements in
> the array
>
> The result is a row-major array and there does not seem to exist a way to
> create a column major array (while Python's buffer protocol explicitly
> allows it).

Is there no way to make the kind of memoryview you want from Python
code, from a raw buffer? For example, forgetting cffi: what if you
start with raw bytes loaded from a file; can you make the memoryview
you want then? If not, it looks like a bug of Python.


Armin

[Laurent Gautier]

Hi Armin,

(more below)

It is very possibly an API / design issue on Python side but I thought I'd ask here first in case I missed something.

I am moving the question to Python-focused forums, while still hoping that I missed something (as Python development cycles means that a possible fix involving an API would take a long time).

 

Armin

--
-- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
---
You received this message because you are subscribed to the Google Groups "python-cffi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-cffi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Armin Rigo]

Hi Laurent,

On 23 September 2018 at 19:38, Laurent Gautier <lgau...@gmail.com> wrote:
> It is very possibly an API / design issue on Python side but I thought I'd
> ask here first in case I missed something.

Generally speaking I don't know all the details of the Python
memoryview from the C API, which I know is half available and half
unavailable from Python. This fact makes me believe it is very
broken, but that's another discussion.

Maybe you can do what you want by involving numpy in the mix. I am
also not familiar with numpy, but try something like: create a numpy
array from the buffer, transpose it, and get the resulting array as a
memoryview again. Maybe you can do that in such a way that the
transpose won't actually copy the data. Or maybe you can directly ask
for a Fortran-style memoryview from the original numpy array.


A bientôt,

Armin.

[Laurent Gautier]

Le dim. 23 sept. 2018 à 14:16, Armin Rigo <armin...@gmail.com> a écrit :

Hi Laurent,

On 23 September 2018 at 19:38, Laurent Gautier <lgau...@gmail.com> wrote:
> It is very possibly an API / design issue on Python side but I thought I'd
> ask here first in case I missed something.

Generally speaking I don't know all the details of the Python
memoryview from the C API, which I know is half available and half
unavailable from Python.  This fact makes me believe it is very
broken, but that's another discussion.

I liked the buffer protocol from the C-API side, but now that I am trying to make it work from the Python side I am finding issues with it.

Maybe you can do what you want by involving numpy in the mix.  I am
also not familiar with numpy, but try something like: create a numpy
array from the buffer, transpose it, and get the resulting array as a
memoryview again.  Maybe you can do that in such a way that the
transpose won't actually copy the data.  Or maybe you can directly ask
for a Fortran-style memoryview from the original numpy array.

Thanks.

There are ways so involve numpy but as I am migrating rpy2 (bridge between Python and R)'s C-API extension to cffi I am trying to port the feature that R arrays (column-major arrays) wrapped by rpy2 implement the buffer protocol, and this allows they seamless use in numpy (or anything else that makes use of that protocol (e.g., Cython finally doing it). cffi is quite nice and rewriting and substancial amount of C-API extension code (~10,000 lines of C source) happened relatively fast.

The buffer protocol cannot be implemented at the Python level (possibly a design oversight, but I have not thought enough about it to be sure), but I was hoping to chain cffi's buffer() and Python's memoryview and memoryview.cast to create buffer protocol-friendly objects with only added burden of making it explicit (rather in implicit in the current rpy2 - numpy.array(r_array.memoryview()) instead of numpy.asarray(r_array).

However, the dependency on numpy is a significantly large one which I'd like to avoid very much. I was hoping to have rpy2 connect to R in "pure Python" mode (cffi and ABI mode for when binary packages or compilation during installation is not possible) or in compiled mode (cffi and API mode with speed bottlenecks as C functions compiled at installation time). I will likely postpone the ABI-only mode to a future rpy2 version (when memoryview and f_contiguous can be specified).

I have also asked about on SO (https://stackoverflow.com/questions/52456805/memoryview-and-column-major-fortran-buffer) and did not receive an answer either so I filed a Python bug report: https://bugs.python.org/issue34778

 

A bientôt,

Armin.