weird crash with cdef classes and C++ STL containers

[David Warde-Farley]

Hello everyone,

So, shoving cdef classes into C++ STL containers *seems* to work (when
compiling with language='c++' of course) but with some strange side
effects. I'm getting quite a strange crash (seg fault), which only
happens the second time I call the function that does it. It seems to
be related to deque, replacing that with a Python deque is fine.
Replacing it with a C++ STL vector still yields the same problem.

https://gist.github.com/30a62791569666f6447b

This crash is happening with Cython 0.17. Anyone know what's going on?

David

[Robert Bradshaw]

Sticking Python objects in c++ containers doesn't work as they don't do reference counting... This should be a compile error.

[David Warde-Farley]

On Wed, Sep 12, 2012 at 11:17 PM, Robert Bradshaw <robe...@gmail.com> wrote:

> Sticking Python objects in c++ containers doesn't work as they don't do
> reference counting... This should be a compile error.

Welp, that explains it. I figured there might be some voodoo going on
that made it possible to refcount cdef classes in STL containers given
that it magically worked (and seems to work without crashing in other
cases).

Thanks!

David

[Robert Bradshaw]

It's storing borrowed references, so it'll work as long as something
else in Python-space keeps it alive (which is, as evidenced, a rather
precarious situation to be in).

- Robert

[David Warde-Farley]

On Thu, Sep 13, 2012 at 12:02 AM, Robert Bradshaw <robe...@gmail.com> wrote:

> It's storing borrowed references, so it'll work as long as something
> else in Python-space keeps it alive (which is, as evidenced, a rather
> precarious situation to be in).

Indeed. I'm confused, though: in the example I've given, all 4 nodes
have references not only to each other but in the method scope of
test_dfs. It's unclear to me how or why they'd be garbage collected
(or decref'd to 0) before test_dfs() finished (since it's a call in
the last line of test_dfs where the crash takes place).

David

[Dag Sverre Seljebotn]

If you want to do this, write an auto_ptr-like class, a class
overloading operator* and operator->, let's call it pyptr, which has a
PyObject* member but does INCREF on construction and DECREF on destruction.

Then simply create std::vector<pyptr> rather than std::vector<PyObject*>.

This *should* work, but to my knowledge there isn't any code available
already, so please report back if you do this and perhaps we can include
the pyptr class with Cython.

Dag Sverre

[Dag Sverre Seljebotn]

In fact, once std::vector<pyptr> works really well, it should be rather
easy to build it into Cython and support "std::vector<object>" as a
syntax for it. Not volunteering though.

Dag Sverre

[Robert Bradshaw]

I was about to suggest exactly this. Syntax is a good question though,
but perhaps "std::vector<object>" would be sufficient.

- Robert

[Bradley Froehle]

A quick search turned up a lot of smart pointer adapters for `PyObject*`s.  For example, http://code.activestate.com/recipes/577985-automatic-python-pyobject-ref-count-management-in-/

-Brad

[Stefan Behnel]

Bradley Froehle, 13.09.2012 18:22:

> On Thursday, September 13, 2012 8:59:43 AM UTC-7, Robert Bradshaw wrote:

>> On Thu, Sep 13, 2012 at 1:58 AM, Dag Sverre Seljebotn wrote:
>>> On 09/13/2012 07:26 AM, David Warde-Farley wrote:

>>>> On Thu, Sep 13, 2012 at 12:02 AM, Robert Bradshaw wrote:
>>>>
>>>>> It's storing borrowed references, so it'll work as long as something
>>>>> else in Python-space keeps it alive (which is, as evidenced, a rather
>>>>> precarious situation to be in).
>>>>
>>>> Indeed. I'm confused, though: in the example I've given, all 4 nodes
>>>> have references not only to each other but in the method scope of
>>>> test_dfs. It's unclear to me how or why they'd be garbage collected
>>>> (or decref'd to 0) before test_dfs() finished (since it's a call in
>>>> the last line of test_dfs where the crash takes place).
>>>
>>> If you want to do this, write an auto_ptr-like class, a class
>>> overloading
>>> operator* and operator->, let's call it pyptr, which has a PyObject*
>>> member but does INCREF on construction and DECREF on destruction.
>>>
>>> Then simply create std::vector<pyptr> rather than
>>> std::vector<PyObject*>.
>>>
>>> This *should* work, but to my knowledge there isn't any code available
>>> already, so please report back if you do this and perhaps we can include
>>> the pyptr class with Cython.
>>
>> I was about to suggest exactly this. Syntax is a good question though,
>> but perhaps "std::vector<object>" would be sufficient.
>

> A quick search turned up a lot of smart pointer adapters for `PyObject*`s.
> For example,
> http://code.activestate.com/recipes/577985-automatic-python-pyobject-ref-count-management-in-/

+1 for supporting this in Cython. Any volunteering to make this work as in
"std::vector<object>" is definitely appreciated.

Stefan