Return tuple of memviews as struct

[jbrock...@gmail.com]

I have a function that currently returns a 3-tuple (ndarray, ndarray, str).  In the .pxd file the return type is 'object' and calls to this function show up as bright yellow when annotated.

Thought I'd try creating a struct and return that instead of a tuple.  Tried (ignoring the str part for part)

ctypedef struct MyStruct:

    ndarray left

    ndarray right

ctypedef struct MyStruct:

    int64_t[::1] left

    int64_t[::1] right

ctypedef struct MyStruct:

    int64_t[::1]* left

    int64_t[::1]* right

ctypedef struct MyStruct:

    int64_t* left

    Py_ssize_t nleft

    int64_t* right

    Py_ssize_t nright

The first three fail at compile time with:

C struct/union member cannot be a Python object,

C struct/union member cannot be a memory view,

Cannot take address of memoryview slice,

respectively.

The last one hasn't failed yet but I haven't updated the calling functions to the new signature so don't have a final verdict.

Is there a more elegant way to do this?

[Stefan Behnel]

Am March 19, 2022 7:54:20 PM UTC schrieb "jbrock...@gmail.com":
>I have a function that currently returns a 3-tuple (ndarray, ndarray,
>str). In the .pxd file the return type is 'object' and calls to this
>function show up as bright yellow when annotated.

Yellow isn't bad, it just means that there is Python object interaction. In this case, that's what you do anyway, you pass back three objects. And that seems the right thing to do.

>Thought I'd try creating a struct and return that instead of a tuple.
>

>Is there a more elegant way to do this?

I'd use a tuple. And also declare the return type as tuple.

Cython 3 has support for PEP 484 tuple item types, so you could declare them explicitly now and thus allow the caller to see exactly what it receives. Not sure if that leads to better code in the current state of the implementation, but it's certainly not wrong to be explicit and might help as we improve Cython.

Stefan

[da-woods]

On 20/03/2022 05:47, Stefan Behnel wrote:
> Thought I'd try creating a struct and return that instead of a tuple.
>> Is there a more elegant way to do this?
> I'd use a tuple. And also declare the return type as tuple.
>
> Cython 3 has support for PEP 484 tuple item types, so you could declare them explicitly now and thus allow the caller to see exactly what it receives. Not sure if that leads to better code in the current state of the implementation, but it's certainly not wrong to be explicit and might help as we improve Cython.
>
> Stefan
>

The current status is:

* A tuple of C types is interpreted as a C tuple (i.e. internally
transformed to a struct in Cython).

* A tuple containing PyObjects is just returned as a normal tuple (since
Cython's C tuple optimization can't currently handle reference-counted
types)

That's currently on the master branch only and not on any alpha releases
(and I think the feature needs a little bit of fixing before it makes it
to an alpha release). So right now I don't think you can do much better
than "just return a tuple"

[Stefan Behnel]

da-woods schrieb am 20.03.22 um 09:29:

> On 20/03/2022 05:47, Stefan Behnel wrote:
>> Thought I'd try creating a struct and return that instead of a tuple.
>>> Is there a more elegant way to do this?
>> I'd use a tuple. And also declare the return type as tuple.
>>
>> Cython 3 has support for PEP 484 tuple item types, so you could declare
>> them explicitly now and thus allow the caller to see exactly what it
>> receives. Not sure if that leads to better code in the current state of
>> the implementation, but it's certainly not wrong to be explicit and might
>> help as we improve Cython.
>

> The current status is:
>
> * A tuple of C types is interpreted as a C tuple (i.e. internally
> transformed to a struct in Cython).
>
> * A tuple containing PyObjects is just returned as a normal tuple (since
> Cython's C tuple optimization can't currently handle reference-counted types)
>
> That's currently on the master branch only and not on any alpha releases
> (and I think the feature needs a little bit of fixing before it makes it to
> an alpha release). So right now I don't think you can do much better than
> "just return a tuple"

… where "cannot do much better" IMHO means "it's a good and efficient
solution" here.

Stefan

[Brock Mendel]

Thanks for the discussion.  It sounds like this isn't a good place to focus my optimization effort for the time being.

--

---
You received this message because you are subscribed to a topic in the Google Groups "cython-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cython-users/oeKJX4Hq6mI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/d950c32d-2ed9-6989-9367-372a3b1ac655%40behnel.de.

[jbrock...@gmail.com]

> Yellow isn't bad, it just means that there is Python object interaction. In this case, that's what you do anyway, you pass back three objects. And that seems the right thing to do.

Request for clarification.  Outside the context of this particular problem, I sometimes look at the annotated html and use "degree of yellowness" as a proxy for "candidate for optimization".  Is that a bad heuristic?

[da-woods]

> Request for clarification.  Outside the context of this particular
> problem, I sometimes look at the annotated html and use "degree of
> yellowness" as a proxy for "candidate for optimization".  Is that a
> bad heuristic?

It's a reasonable heuristic (and it's what it's there for...) but it has
limitations. The most obvious is that you should multiply it by how
often the line is called. I think the algorithm for "degree of
yellowness" is always relatively crude too
https://github.com/cython/cython/blob/master/Cython/Compiler/Annotate.py#L259-L261
(really just a count of different "types" of call)