Return tuple of memviews as struct

21 views
Skip to first unread message

jbrock...@gmail.com

unread,
Mar 19, 2022, 3:54:20 PMMar 19
to cython-users
I have a function that currently returns a 3-tuple (ndarray, ndarray, str).  In the .pxd file the return type is 'object' and calls to this function show up as bright yellow when annotated.

Thought I'd try creating a struct and return that instead of a tuple.  Tried (ignoring the str part for part)

ctypedef struct MyStruct:
    ndarray left
    ndarray right

ctypedef struct MyStruct:
    int64_t[::1] left
    int64_t[::1] right

ctypedef struct MyStruct:
    int64_t[::1]* left
    int64_t[::1]* right

ctypedef struct MyStruct:
    int64_t* left
    Py_ssize_t nleft
    int64_t* right
    Py_ssize_t nright

The first three fail at compile time with:
C struct/union member cannot be a Python object,
C struct/union member cannot be a memory view,
Cannot take address of memoryview slice,
respectively.

The last one hasn't failed yet but I haven't updated the calling functions to the new signature so don't have a final verdict.

Is there a more elegant way to do this?

Stefan Behnel

unread,
Mar 20, 2022, 1:47:41 AMMar 20
to cython...@googlegroups.com
Am March 19, 2022 7:54:20 PM UTC schrieb "jbrock...@gmail.com":
>I have a function that currently returns a 3-tuple (ndarray, ndarray,
>str). In the .pxd file the return type is 'object' and calls to this
>function show up as bright yellow when annotated.

Yellow isn't bad, it just means that there is Python object interaction. In this case, that's what you do anyway, you pass back three objects. And that seems the right thing to do.


>Thought I'd try creating a struct and return that instead of a tuple.
>
>Is there a more elegant way to do this?

I'd use a tuple. And also declare the return type as tuple.

Cython 3 has support for PEP 484 tuple item types, so you could declare them explicitly now and thus allow the caller to see exactly what it receives. Not sure if that leads to better code in the current state of the implementation, but it's certainly not wrong to be explicit and might help as we improve Cython.

Stefan

da-woods

unread,
Mar 20, 2022, 4:31:40 AMMar 20
to cython...@googlegroups.com
On 20/03/2022 05:47, Stefan Behnel wrote:
> Thought I'd try creating a struct and return that instead of a tuple.
>> Is there a more elegant way to do this?
> I'd use a tuple. And also declare the return type as tuple.
>
> Cython 3 has support for PEP 484 tuple item types, so you could declare them explicitly now and thus allow the caller to see exactly what it receives. Not sure if that leads to better code in the current state of the implementation, but it's certainly not wrong to be explicit and might help as we improve Cython.
>
> Stefan
>
The current status is:

* A tuple of C types is interpreted as a C tuple (i.e. internally
transformed to a struct in Cython).

* A tuple containing PyObjects is just returned as a normal tuple (since
Cython's C tuple optimization can't currently handle reference-counted
types)

That's currently on the master branch only and not on any alpha releases
(and I think the feature needs a little bit of fixing before it makes it
to an alpha release). So right now I don't think you can do much better
than "just return a tuple"

Stefan Behnel

unread,
Mar 20, 2022, 4:37:39 AMMar 20
to cython...@googlegroups.com
da-woods schrieb am 20.03.22 um 09:29:
> On 20/03/2022 05:47, Stefan Behnel wrote:
>> Thought I'd try creating a struct and return that instead of a tuple.
>>> Is there a more elegant way to do this?
>> I'd use a tuple. And also declare the return type as tuple.
>>
>> Cython 3 has support for PEP 484 tuple item types, so you could declare
>> them explicitly now and thus allow the caller to see exactly what it
>> receives. Not sure if that leads to better code in the current state of
>> the implementation, but it's certainly not wrong to be explicit and might
>> help as we improve Cython.
>
> The current status is:
>
> * A tuple of C types is interpreted as a C tuple (i.e. internally
> transformed to a struct in Cython).
>
> * A tuple containing PyObjects is just returned as a normal tuple (since
> Cython's C tuple optimization can't currently handle reference-counted types)
>
> That's currently on the master branch only and not on any alpha releases
> (and I think the feature needs a little bit of fixing before it makes it to
> an alpha release). So right now I don't think you can do much better than
> "just return a tuple"

… where "cannot do much better" IMHO means "it's a good and efficient
solution" here.

Stefan

Brock Mendel

unread,
Mar 20, 2022, 10:55:45 AMMar 20
to cython...@googlegroups.com
Thanks for the discussion.  It sounds like this isn't a good place to focus my optimization effort for the time being.

--

---
You received this message because you are subscribed to a topic in the Google Groups "cython-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cython-users/oeKJX4Hq6mI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/d950c32d-2ed9-6989-9367-372a3b1ac655%40behnel.de.

jbrock...@gmail.com

unread,
Mar 22, 2022, 2:14:04 PMMar 22
to cython-users
> Yellow isn't bad, it just means that there is Python object interaction. In this case, that's what you do anyway, you pass back three objects. And that seems the right thing to do.

Request for clarification.  Outside the context of this particular problem, I sometimes look at the annotated html and use "degree of yellowness" as a proxy for "candidate for optimization".  Is that a bad heuristic?

da-woods

unread,
Mar 22, 2022, 3:44:57 PMMar 22
to cython...@googlegroups.com

> Request for clarification.  Outside the context of this particular
> problem, I sometimes look at the annotated html and use "degree of
> yellowness" as a proxy for "candidate for optimization".  Is that a
> bad heuristic?

It's a reasonable heuristic (and it's what it's there for...) but it has
limitations. The most obvious is that you should multiply it by how
often the line is called. I think the algorithm for "degree of
yellowness" is always relatively crude too
https://github.com/cython/cython/blob/master/Cython/Compiler/Annotate.py#L259-L261
(really just a count of different "types" of call)

Reply all
Reply to author
Forward
0 new messages