Garbage collection problems in cython

[Ian Bell]

I have pretty much finished an extension type that wraps a double* array in Cython that is meant for super-lightweight element-wise operations and instantiation.  The code is part of my repo at: http://sourceforge.net/p/pdsim/git/ci/83bb88f5fda892d3f5d1e1bb8a76bf5dbba8c6e7/tree/PDSim/misc/datatypes.pyx

The element-wise operations work and are lightning fast (finally), but I am having problems with the reference counting.  I am doing lots of these operations with these arraym objects, but it seems like they are not getting garbage collected properly because my memory usage zooms up at about 10 MB/s (ouch!).  But after the program completes I get my memory back - so it seems we are not leaking memory at least.  But I need to be able to run for more than 2 minutes so I need to figure out how to fix the reference counting.

I would like to see this code make its way into the repo since I am pretty sure it would be useful to a lot of people, I think the garbage collecting is one of the last remaining problems with my extension type.

Ian

[Robert Bradshaw]

Are you sure you're not hanging on to them somewhere? That's the only
way I could see them not getting garbage collected (though a pointer
to the code you're running that runs out of memory would be helpful
too).

- Robert

[Ian Bell]

Maybe I am hanging onto them in my container class - Im not entirely sure though.

I put a few arraym into a container class with this PXD file snippet:

cdef class CVArrays(object):
    cdef list array_list #A list of strings with the names of the arrays ['T','p',....]
   
    #Storage arrays
    cdef public arraym T,p,h,rho,V,dV,cp,cv,m,v,dpdT_constV,Q,xL,dudxL
   
    #Property derivative arrays
    cdef public arraym drhodtheta, dTdtheta, dmdtheta, dxLdtheta, summerdm, summerdT, summerdxL, property_derivs

and I try to kill the arraym instances when I am destructing the CVArrays instance by doing

cpdef free_all(self):
        """
        Free all the arrays allocated
        """
        for array_name in self.array_list:
            if hasattr(self,array_name) and getattr(self,array_name) is not None:
                (<arraym>getattr(self,array_name)).dealloc()
                delattr(self,array_name)

and when I check gc.get_referents on the CVArrays instance it still sees all the arrays, though their memory has been deallocated.  So perhaps the problem is elsewhere - I'm at my wits end now.

Do you have recommendations of means of debugging garbage collection?

[Dag Sverre Seljebotn]

Try to assign None rather than using delattr.

Putting a print statement in the __dealloc__(self) method can be
instructive sometimes (though the presence of the __dealloc__ can change
the behaviour too...).

Dag Sverre

[Robert Bradshaw]

Not other than using the standard Python tools, and perhaps print
statements and/or counters in your __cinit__ and __dealloc__ methods
to make sure things are happening as you expect.

I wonder, though, if you're making things way more complicated than
you need. For example, do you need a free_all method, or can you just
throw the CVArrays object away (in which case they'll get cleaned up
automatically, and the dealloc call is totally redundant as well)?

- Robert

[Ian Bell]

Ok, so pie-on-the-face time.  It seems it is actually a memory leak from another library that is linked into my project.  It looked for a long time like a cython-related issue, but as usual, its the fault of the programmer, not Cython.

In any case, I am going to re-write the class a little bit to clean things up based on your suggestions!

Sorry for taking your time with this.

Ian