Re: [cython-users] C++ Vector is not working well for Python object.

[Chris Barker - NOAA Federal]

On Thu, Feb 7, 2013 at 8:12 AM, Debarshi Goswami
<debarshi...@gmail.com> wrote:
> I want to use C++ vector for storing some Python object and later iterate,
> add and delete those as per requirement.

uum, why?

A Python list is very similar to a C++ vector, but it will interact
with python reference counting, etc. properly without your needing to
do anything.

-Chris



--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Robert Bradshaw]

On Sat, Feb 9, 2013 at 5:54 AM, Debarshi Goswami
<debarshi...@gmail.com> wrote:
> Chris,
> I wanted to use a C++ template class in Cython which acts similar to vector
> with some extra functionality. I am working in a tool, which supports
> multiple languages like, C++, Java, C# and Python is one of those.
>
> This tool has some core classes, which are used by all other languages and
> those classes are written in C++ using templates. All languages refers those
> classes and it increases maintainability (Obviously I don't want multiple
> implementation of core algorithm). I was just trying to know how template
> behaves with Python object in Cython code.

If you really want to store Python objects in C++ containers, you need
to worry about reference counting, as placement in a vector, etc. will
not keep the item alive. There are two main approaches to do this:

(1) Store every object you put in your C++ container somewhere else as
well, e.g. keep a list to the side that you add items to/remove items
from as they are added to/removed from your vector.

(2) Write a simple C++ smart pointer reference class that increfs
items on assignment/creation and decrefs on deletion. Store wrapped
Python objects in your containers.

On this note, we should probably dis-allow using python objects as
template parameters...

> On Thursday, 7 February 2013 23:44:54 UTC+5:30, Chris Barker - NOAA Federal

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[John Tyree]

On this note, we should probably dis-allow using python objects as
template parameters...

If and when this happens, make sure the error message mentions what the issue is. This will be a confusing and very arbitrary error for many, given the general understanding of templates and apparent lack of familiarity with python's reference counting. 

-John

[Chris Barker - NOAA Federal]

On Sat, Feb 9, 2013 at 11:38 AM, Robert Bradshaw <robe...@gmail.com> wrote:

> (2) Write a simple C++ smart pointer reference class that increfs
> items on assignment/creation and decrefs on deletion. Store wrapped
> Python objects in your containers.

> On this note, we should probably dis-allow using python objects as
> template parameters...

Showing my C++ ignorance here:

Could one write such a "simple smart pointer reference class" that
could be used to wrap python object pointers and used with arbitrary
C++ containers? If so, that would be a great addition to Cython,
particularly if Cython dis-allows python objects as template
parameters. i.e disallow it, but offer an out-of-the-box alternative.

[Chris Barker - NOAA Federal]

On Sat, Feb 9, 2013 at 5:54 AM, Debarshi Goswami
<debarshi...@gmail.com> wrote:

> I wanted to use a C++ template class in Cython which acts similar to vector
> with some extra functionality. I am working in a tool, which supports
> multiple languages like, C++, Java, C# and Python is one of those.

OK -- makes sense, you were using vector as an example test case.

> This tool has some core classes, which are used by all other languages and
> those classes are written in C++ using templates. All languages refers those
> classes and it increases maintainability (Obviously I don't want multiple
> implementation of core algorithm).

I'm still a bit confused -- Python objects are Python-specific, so if
you store them in a C+ container your C++ code that works with those
objects would have to know about Python objects. Or is the container
class itself the entire "core algorithm"?

[Sturla Molden]

On 09.02.2013 20:38, Robert Bradshaw wrote:

> On this note, we should probably dis-allow using python objects as
> template parameters...

It's inherently dangerous, so please disallow it.


Sturla

[Sturla Molden]

On 11.02.2013 17:29, Chris Barker - NOAA Federal wrote:

> I'm still a bit confused -- Python objects are Python-specific, so if
> you store them in a C+ container your C++ code that works with those
> objects would have to know about Python objects. Or is the container
> class itself the entire "core algorithm"?

Python objects are allocated by Python, so we could e.g. have

std::vector<PyObject*>

but never

std::vector<PyObject>

However, in the first case the C++ vector will keep the object alive,
i.e. that object that the PyObject* pointer is pointing to. And that
will require reference counting. That is one of the reasons it is
recommended to use CXX and not the Python C API directly when writing
Python extensions manually in C++.

What we need then is a small C++ class that stores a PyObject*
internally and keeps track of the reference counting. The container
itsef (e.g. std::vector<T>) does not need to know about Python objects.
But the wrapper class for the PyObject* will use its constructor and
destructor to incref and decref. Incref and decref will be called
deterministically (and thus in the correct place) because of the RAII
nature of C++. So the C++ container does not need to know about
references. But whenever a Python object is used as a C++ template
parameter, Cython should insert the C++ wrapper class instead. But until
then, using Python objects as template parameters should probably be
disallowed.


Sturla

[Robert Bradshaw]

On Tue, Feb 12, 2013 at 5:11 AM, Sturla Molden <stu...@molden.no> wrote:
> On 11.02.2013 17:29, Chris Barker - NOAA Federal wrote:
>
>> I'm still a bit confused -- Python objects are Python-specific, so if
>> you store them in a C+ container your C++ code that works with those
>> objects would have to know about Python objects. Or is the container
>> class itself the entire "core algorithm"?
>
>
> Python objects are allocated by Python, so we could e.g. have
>
> std::vector<PyObject*>
>
> but never
>
> std::vector<PyObject>
>
> However, in the first case the C++ vector will keep the object alive, i.e.
> that object that the PyObject* pointer is pointing to. And that will require
> reference counting. That is one of the reasons it is recommended to use CXX
> and not the Python C API directly when writing Python extensions manually in
> C++.
>
> What we need then is a small C++ class that stores a PyObject* internally
> and keeps track of the reference counting. The container itsef (e.g.
> std::vector<T>) does not need to know about Python objects. But the wrapper
> class for the PyObject* will use its constructor and destructor to incref
> and decref. Incref and decref will be called deterministically (and thus in
> the correct place) because of the RAII nature of C++. So the C++ container
> does not need to know about references.

Yep, this was option (2). I've even implemented such a class myself
(as I'm sure have several others.)

> But whenever a Python object is used
> as a C++ template parameter, Cython should insert the C++ wrapper class
> instead.

We could provide such a class explicitly in the Python API, e.g.

from cython import PyRef

cdef vector<PyRef[Foo]> v
v.push_back(PyRef(Foo())
cdef Foo foo = v.at(0).Get()

which would do the right thing. We would have to add some support in
the compiler for this, at the very least avoiding a warning for an
object "template" parameter. We could consider adding special syntax
to deal with this, but I don't see how to make it completely
transparent without "cdef Foo" becoming ambiguous (e.g. in a method
signature, though one can't rule out being able to do fancy-enough
things with assignment operators...).

On this flip side, there's the idea of adding syntax for a borrowed
reference, e.g.

cdef Foo f = ...
cdef Foo* borrowed_foo = &f # does not keep f alive, just like C
cdef Foo f2 = f[0]

though this does run into sticky issues, like having to dis-allow.

cdef Foo* borrowed_foo = malloc(???)

> But until then, using Python objects as template parameters should
> probably be disallowed.

Yes.

- Robert