Question about Memory management boundaries for cython embedded in C++ app

[RJ Marcus]

Hello Cython devs, 

I am working on a project where we have a main C++ application that embeds a python interpreter and makes calls into cython/python code. 

We have a memory leak inside a particular part of our cython. I have made a bare bones example of what I think the problem is (available here: https://github.com/neontty/cython_memory_example1) . 

Primary Question:

Essentially we instantiate a large struct on the heap in C++ that contains a vector of ints and pass that struct by pointer into cython where calculations are done. The vector of ints needs to be resized before I can assign the results from cython via memoryviews. 

Is that illegal in terms of memory management? Am I allowed to call std::vector.resize() in that context from cython? Is the memory ownership somehow changing when the vector gets reallocated?

Note: in that github repo I posted with the example code, I can't the "mismatched free" I was seeing in valgrind (on my work computer) to show up, but in our original project we had a "mismatched free" that pointed exactly to the "vector.resize()" call. Hopefully the code is still useful to you for understanding the context of the code I'm analyzing.

Secondary Question:
Does anyone have a preferred method for debugging memory leaks for cython/python embedded in C++ ? Currently I'm using valgrind with the python suppressions file, but we also use gperftools on our C++ apps and I'd prefer to use that if possible on the cython also. (Tried compiling my cython extensions with -ltcmalloc and recompiling the python interpreter, but it still seemed like it gave garbage output). 

Thank you so much for reading this! Any help is appreciated.

Richard

[Stefan Behnel]

Am 1. August 2021 03:44:32 MESZ schrieb RJ Marcus:
>Hello Cython devs,
>
>I am working on a project where we have a main C++ application that embeds
>a python interpreter and makes calls into cython/python code.
>
>We have a memory leak inside a particular part of our cython. I have made a
>bare bones example of what I think the problem is (available here:
>https://github.com/neontty/cython_memory_example1) .
>
>

>*Primary Question:*

>
>Essentially we instantiate a large struct on the heap in C++ that contains
>a vector of ints and pass that struct by pointer into cython where
>calculations are done. The vector of ints needs to be resized before I can
>assign the results from cython via memoryviews.
>
>Is that illegal in terms of memory management? Am I allowed to call
>std::vector.resize() in that context from cython? Is the memory ownership
>somehow changing when the vector gets reallocated?

No, it should generally be fine, but you might need to use Cython 3.0 to make it work. IIRC, 0.29 does not automatically deallocate C++ objects in structs. (You had to call "del" on it yourself.)


>*Secondary Question:*

>Does anyone have a preferred method for debugging memory leaks for
>cython/python embedded in C++ ? Currently I'm using valgrind with the
>python suppressions file, but we also use gperftools on our C++ apps and
>I'd prefer to use that if possible on the cython also. (Tried compiling my
>cython extensions with -ltcmalloc and recompiling the python interpreter,
>but it still seemed like it gave garbage output).

Never tried that. I personally use valgrind.

Stefan

[D Woods]

>>Essentially we instantiate a large struct on the heap in C++ that contains
>>a vector of ints and pass that struct by pointer into cython where
>>calculations are done. The vector of ints needs to be resized before I can
>>assign the results from cython via memoryviews.
>>
>>Is that illegal in terms of memory management? Am I allowed to call
>>std::vector.resize() in that context from cython? Is the memory ownership
>>somehow changing when the vector gets reallocated?

> No, it should generally be fine, but you might need to use Cython 3.0 to make it work. IIRC, 0.29 does not automatically deallocate C++ objects in structs. (You >had to call "del" on it yourself.)

To clarify this point, the issue was structs inside a cdef class:

cdef extern from "somewhere":

    cdef struct S:

        cdef vector[int] vec

cdef class C:

   cdef S s_inst

(In Cython 0.29) the destructor for S will not get called, and thus the destructor for vec will not get called. The solution would generally either be to update Cython or to tell Cython that S is a `cppclass` instead of a struct.

----------------------------

In your bare-bones example, I'm not convinced that your "error" cases clean up correctly

[Richard Marcus]

Stefan and D Woods, 

Thank you for responding so quickly! Your explanations are well received and I will first try changing the object into a class (failing that, I will try cython 3.0). 

D Woods, would you mind elaborating on which error cases? The bare-bones example is not perfect or an exact copy of my production code, but I'd like to understand in case there is more knowledge for me to gain here.

Cheers,

RJ

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/614dffc7-7e0a-4945-a913-99aee2bfcda2n%40googlegroups.com.

[D Woods]

> D Woods, would you mind elaborating on which error cases?

Sorry! On a closer look they do appear to be handled correctly. I'd got it into my head that you returned after the two "PyErr_PrintEx" (and thus didn't "delete instance"). But I think I read what I was expecting to see rather was there rather than what was actually there.

Sorry for the confusion.

David