Fastest way of converting a C array into a Python list

[Simon King]

Hi!

I am sure that the following problem has a solution that is well-known
- but not well-known to me, and also I did not succeed in googling it.

Background is #12029.

In the following, self is a ClonableIntArray, but I guess the problem
is more general. self._list is some "int *" array of known length
self._len. The aim is to convert self._list into a Python list, as
quickly as possible.

Is there a ready-made function that would do such conversion?

I did this, which works fine:
cdef int i
cdef list L = []
for i from 0<=i<self._len:
L.append(self._list[i])
return L

But Florent was suggesting that it might be faster to create an
"empty" list of length self._len using PyList_New, and to put the
items onto that list by PyList_SetItem, rather than appending them.

First question: Could PyList_SET_ITEM (for a list of known length)
really be faster than appending to the list?

Second question: I know that PyList_SET_ITEM is only borrowing a
reference. Hence, I am not surprised that I got segfaults in the first
place. But why am I still getting segfaults with the following code,
where I manually add a reference?
cdef int i, o
cdef list L = PyList_New(self._len)
for i from 0<=i<self._len:
o = self._list[i]
Py_INCREF(o)
PyList_SET_ITEM(L,i,o)
return L

More precisely, it works (or seems to work) on ClonableIntArrays of
size 100, but not on a list of size 500.

Best regards,
Simon

[Florent Hivert]

For the record, I think I answered all of this on #12029.

Cheers,

Florent

[Simon King]

On 14 Nov., 23:54, Simon King <simon.k...@uni-jena.de> wrote:
> But why am I still getting segfaults with the following code,
> where I manually add a reference?
>         cdef int i, o
>         cdef list L = PyList_New(self._len)
>         for  i from 0<=i<self._len:
>             o = self._list[i]
>             Py_INCREF(o)
>             PyList_SET_ITEM(L,i,o)
>         return L

While I wrote this, Florent was producing a quite similar code, with
small differences: He did
cdef object o
and
o = PyInt_FromLong(self._list[i])
The rest is the same

Why does Florent's code work and mine doesn't?

Best regards,
Simon

[Simon King]

Hi Florent,

On 15 Nov., 00:05, Florent Hivert <Florent.Hiv...@lri.fr> wrote:
> For the record, I think I answered all of this on #12029.

Your code seems to work, but you didn't answer my question. Since the
only difference to my code is your use of PyInt_FromLong, I don't see
yet why it works, while my code doesn't.

I was using Py_INCREF on an int. Should PyINCREF only be used on an
object, perhaps?

Cheers,
Simon

[Florent Hivert]

As far as I understand in

cdef int i, o
Py_INCREF(o)

Since o is a C int, the call

Py_INCREF(o)

convert o to a Python object that is create a o1, increment the ref-counting
of o1. Then

PyList_SET_ITEM(L,i,o)

create another o2 and pass it to PyList_SET_ITEM

Cheers,

Florent

[Florent Hivert]

Yes ! See my other answer. We should synchonize on IRC next time ;-)

I'm off to bed.

Florent