using cdef class to expose c++ classes to python and issues with __dealloc__

[Michael Hogg]

Hi,

I have an octree code written in C++ and am trying to expose it to Python. The exposure is fairly minimal. I want to use Python to pass the information required to setup the octree, and after it is setup to be able to retrieve information from it (but not change it after it has been set up) This keeps all the number crunching in C++.

Because I just want to retrieve info from existing instances of the C++ classes (cOctree and cOctNode), I have just created corresponding cdef classes (PyOctree and PyOctnode) and then used a global function (PyOctnode_Init) to set the pointer of the cdef class to the C++ class instance as suggested in this post: http://stackoverflow.com/questions/12204441/passing-c-pointer-as-argument-into-cython-function.

This works, however after I added the __dealloc__ function to PyOctNode, I started getting crashes. I suspect that this may be because I am using 'property' to expose the C++ class instances and the objects go out of scope. I am not sure if this is the correct way of doing this, or if there is a better way.

Any help appreciated.

Cheers,

Michael 

The contents of cython file octree.pyx:

cdef extern from "cOctree.h":

    cdef cppclass cOctNode:

        double size

        int level

        int nid

        vector[double] position

        vector[cOctNode] branches

        vector[int] data

        int numPolys()

        bint isLeafNode()

    cdef cppclass cOctree:

        cOctree(vector[vector[double]] vertexCoords3D, vector[vector[int]] polyConnectivity)

        int numPolys()

        cOctNode root

cdef class PyOctree:

    cdef cOctree *thisptr

    def __cinit__(self,double[:,::1] _vertexCoords3D, int[:,::1] _polyConnectivity):

        cdef int i, j

        cdef vector[double] coords

        cdef vector[vector[double]] vertexCoords3D

        coords.resize(3)

        for i in range(_vertexCoords3D.shape[0]):

            for j in range(3):

                coords[j] = _vertexCoords3D[i,j]

            vertexCoords3D.push_back(coords)

        cdef vector[int] connect

        cdef vector[vector[int]] polyConnectivity

        connect.resize(3)

        for i in range(_polyConnectivity.shape[0]):

            for j in range(3):

                connect[j] = _polyConnectivity[i,j]

            polyConnectivity.push_back(connect)

        self.thisptr = new cOctree(vertexCoords3D,polyConnectivity)

    def __dealloc__(self):

       del self.thisptr

    cpdef int numPolys(self):

        return self.thisptr.numPolys()

    property root:

        def __get__(self):

            cdef cOctNode *node = &self.thisptr.root

            return PyOctnode_Init(node)

cdef class PyOctnode:

    cdef cOctNode *thisptr

    def __cinit__(self):

        # self.thisptr will be set by global function PyOctnode_Init to point

        # to an existing cOctNode object

        self.thisptr = NULL

    def __init__(self):

        pass

    property isLeaf:

        def __get__(self):

        return self.thisptr.isLeafNode()

    property branches:

        def __get__(self):

            branches = []

            cdef int numBranches = self.thisptr.branches.size()

            cdef int i

            cdef cOctNode *node = NULL

            for i in range(numBranches):

                node = &self.thisptr.branches[i]

            branches.append(PyOctnode_Init(node))

            return branches

    property polyList:

        def __get__(self):

            cdef list polyList = []

            cdef int numPolys = self.thisptr.numPolys()

            cdef int i

            for i in range(numPolys):

                polyList.append(self.thisptr.data[i])

            return polyList

    property level:

        def __get__(self):

            return self.thisptr.level

    property nid:

        def __get__(self):

            return self.thisptr.nid

    property numPolys:

        def __get__(self):

            return self.thisptr.numPolys()

    property size:

        def __get__(self):

            return self.thisptr.size

    property position:

        def __get__(self):

            cdef int dims = self.thisptr.position.size()

            cdef int i

            cdef np.ndarray[float64,ndim=1] position = np.zeros(3,dtype=np.float64)

            for i in range(dims):

                position[i] = self.thisptr.position[i]

            return position

        def __dealloc__(self):

            print 'PyOctNode __dealloc__(): This will cause a crash'

            #del self.thisptr     

cdef PyOctnode_Init(cOctNode *node):

    result = PyOctnode()

    result.thisptr = node

    return result

[Yury V. Zaytsev]

On Tue, 2014-11-25 at 22:18 -0800, Michael Hogg wrote:
>
> This works, however after I added the __dealloc__ function to
> PyOctNode, I started getting crashes. I suspect that this may be
> because I am using 'property' to expose the C++ class instances and
> the objects go out of scope. I am not sure if this is the correct way
> of doing this, or if there is a better way.

If I understood you correctly, you've created a factory that makes
instances of PyOctNode with the internal pointer set to a specific
instance of the cOctNode class (C++), the properties of which you want
to access.

In this case, why would you need __dealloc__ at all? The C++ code that
creates and manages cOctNodes should take care of deallocation, and make
sure that the objects remain alive until the Python side doesn't need
them anymore. If my assertions are correct and you free these objects in
__dealloc__, of course you're gonna get crashes...

--
Sincerely yours,
Yury V. Zaytsev

[Michael Hogg]

In this case, why would you need __dealloc__ at all? The C++ code that
creates and manages cOctNodes should take care of deallocation, and make
sure that the objects remain alive until the Python side doesn't need
them anymore. If my assertions are correct and you free these objects in
__dealloc__, of course you're gonna get crashes...

Hi Yury,

I think you are right. The cOctNodes are managed by the cOctree. As long as I have a __dealloc__ in PyOctree that deletes the pointer to the cOctree, I should be fine.

Thanks for your help.

Michael

[Brett Calcott]

If you want this to be safe for public consumption, you might want to add an Octree ref into the Nodes too. 

Consider the following:

tree = Octree()

root = tree.root()

del tree

# what will this do (boom!)?

root.position

I think this will fix it:

cdef class cOctNode:

  cdef object tree

  def __cinit__(self, tree):

     self.tree = tree

--

Brett

[Michael Hogg]

Consider the following:

tree = Octree()

root = tree.root()

del tree

# what will this do (boom!)?

root.position

I think this will fix it:

cdef class cOctNode:

  cdef object tree

  def __cinit__(self, tree):

     self.tree = tree

Hi Brett,

I tested out what you suggested. When I tried:

tree = PyOctree()

root = tree.root

del tree

root.position

I don't get a crash. The __dealloc__ function of the PyOctree is called, which in turn calls the destructor of the (C++) cOctree, which then destroys all the cOctNodes (including root). 

But, as I said, I don't get a crash, and I can still access root and all its branches. I am guessing that this is just because the memory hasn't been overwritten yet, and when it does it, it will eventually go boom as you suggest.

In regards to your suggestion, if I do:

cdef class PyOctNode:

  cdef object tree

  def __cinit__(self, tree):

     self.tree = tree

then even if tree is deleted as above, then self.tree will still exist and point to the cOctree. Is this how this works?

Because my cOctree is the one that destroys all the cOctnodes, if the PyOctree gets deleted, then there will be no way of then releasing the memory for the copy of root because PyOctNode doesn't do anything in the __dealloc__ function. This probably my main problem, and related to the issue that you have raised. Any ideas?

Cheers,

Michael

[Brett Calcott]

On Sun, Nov 30, 2014 at 11:57 PM, Michael Hogg <michael.chri...@gmail.com> wrote:

Because my cOctree is the one that destroys all the cOctnodes, if the PyOctree gets deleted, then there will be no way of then releasing the memory for the copy of root because PyOctNode doesn't do anything in the __dealloc__ function. This probably my main problem, and related to the issue that you have raised. Any ideas?

It will work fine. The "object tree" is a python object with a reference count. When the PyOctNode is deleted, it will decrement the reference count to the PyOctTree object. If there are no more references, then it will be deleted and the __dealloc__ will be called.

If you wonder how all this works, it is good to a) look at the generated c/c++ code, or just build a simple case with lots of print statements. I've attached 3 files that demonstrate the idea...

Brett