Is it possible to detect memory leaks with valgrind ?

[S.Golge]

Hello Cython-users,

I have been struggling with -potential- memory leak detection within my cython codes. I would like to use valgrind to detect leaks, if there is any. So I followed the guidelines provided in [1] and uncommented the recommended lines (valgrind-python.supp is attached).  For illustration, I did a simple test to compare memory leak detection of valgrind between a C code and Cython (where a Cython module is explicitly imported).

Here is the naive C code:

      #include <stdlib.h>

      int main(){

      double *x; 

      x= malloc(1000*sizeof(double));

      //free(x); //When uncommented the error disappears confirming that memory is freed

      return 0;}

Compiled it with gcc 4.8.3 on x64 SL7:

     gcc -g -o mem_leak mem_leak.c

After that, I ran valgrind with:

      valgrind --tool=memcheck --leak-check=full ./mem_leak

Provided the following  output as expected complaining that I have memory leak if 8,000 bytes:

.....

.....

==16119== HEAP SUMMARY:

==16119==     in use at exit: 8,000 bytes in 1 blocks

==16119==   total heap usage: 1 allocs, 0 frees, 8,000 bytes allocated

==16119== 

==16119== Searching for pointers to 1 not-freed blocks

==16119== Checked 75,320 bytes

==16119== 

==16119== 8,000 bytes in 1 blocks are definitely lost in loss record 1 of 1

==16119==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)

==16119==    by 0x400541: main (mem_leak.c:5)

==16119== 

==16119== LEAK SUMMARY:

==16119==    definitely lost: 8,000 bytes in 1 blocks

==16119==    indirectly lost: 0 bytes in 0 blocks

==16119==      possibly lost: 0 bytes in 0 blocks

==16119==    still reachable: 0 bytes in 0 blocks

==16119==         suppressed: 0 bytes in 0 blocks

==16119== 

==16119== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)

--16119-- 

--16119-- used_suppression:      2 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a /usr/lib64/valgrind/default.supp:1296

==16119== 

==16119== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)

Then I use a malloc in a Cython code called, which is called from a python code:

#!/usr/bin/python

from cymem_leak import testmain

testmain()

And below cython code is of course compiled with option -g as well added into the setup.py file:

 

# -*- coding: utf-8 -*-

from libc.stdlib cimport malloc, free,abort

def testmain():

  return cymain()

cdef cymain(): 

  cdef double *x

  x= <double*> malloc(sizeof(double) * 1000)

  #free(x) #Whether commented or uncommented, valgrind output does not change at all

  return 0;

After compilation I ran valgrind for this cython code with:

valgrind --tool=memcheck --leak-check=full --suppressions=valgrind-python.supp python ./pymem_leak.py

One would expect that I should get a similar error as I did with C, but, I do not get any specific error related to this memory leak only a long list of errors (full output is attached):

....

.....

==16190== LEAK SUMMARY:

==16190==    definitely lost: 0 bytes in 0 blocks

==16190==    indirectly lost: 0 bytes in 0 blocks

==16190==      possibly lost: 280,279 bytes in 1,025 blocks

==16190==    still reachable: 591,289 bytes in 4,288 blocks

==16190==         suppressed: 0 bytes in 0 blocks

==16190== Reachable blocks (those to which a pointer was found) are not shown.

==16190== To see them, rerun with: --leak-check=full --show-leak-kinds=all

==16190== 

==16190== For counts of detected and suppressed errors, rerun with: -v

==16190== ERROR SUMMARY: 300 errors from 300 contexts (suppressed: 2 from 2)

The bottomline is, either I am doing something incomplete, or it seems it is not possible to identify the memoryleaks in Cython by using valgrind. I would greatly appreciate your help. Thank you

S. Golge

 

[Daπid]

On 21 September 2015 at 23:43, S.Golge <serka...@gmail.com> wrote:

I have been struggling with -potential- memory leak detection within my cython codes. I would like to use valgrind to detect leaks, if there is any.

Give cppcheck a try. It is a static analyser, so not as reliable as valgrind, but it is quite sensitive and easy to use.


http://cppcheck.sourceforge.net/

[S.Golge]

David,

Thanks for recommendation. I will check that one.

However, I am still hoping that someone familiar with how to use valgrind and cython will be kind enough to answer my question. 

[houbaastef .]

Looks very interesting, but for good static analysis, the Python C API should be described in a .cfg file (try cppcheck --force --check-library --enable=information some_cython_extension.c).

Has anyone tried to write such a configuration file ?

Regards,

Stephane

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[S.Golge]

Stephane,

As you noticed, without the proper config file cppcheck will not be able to check the potential errors in cython extension properly. A google search and github did not produce any result for a cfg file.

I have done a through search within this user group and I noticed that long-time users have experience on how to use Valgrind with cython, which seems like the best option. Nevertheless, following the guideline did not help much. I am certain that the Cython developers have some methods to properly debug memory related errors.  Let us wait if any one of them is willing to share their expertise on that. 

[Daπid]

On 22 September 2015 at 18:48, houbaastef . <stef....@gmail.com> wrote:

Looks very interesting, but for good static analysis, the Python C API should be described in a .cfg file (try cppcheck --force --check-library --enable=information some_cython_extension.c).

I have tested it on my own code, as well as Numpy, Scipy, and CPython. In my code, I am not getting any noise, and found a bug in Cython's generated code for memoryviews. In Scipy, I found three lurking memory leaks, and no false positives. Numpy, being a very complex beast, is giving me 20 errors, of which all the ones I have checked so far are false positives; but I think 20 errors is a manageable number. CPython comes out clean.

It is not perfect, but it is worth a shot and it is easier to use than Valgrind, that is, absolutely, the definitive answer if you have a running case of memory leak.

/David.

[S.Golge]

David,

I have 2 questions if you have time to look for:

1) Do you have a configuration file so that cppcheck knows how to treat Python/C API objects ? How did you manage it ? Because this is what I get:

Checking cymem_leak.c...

Checking cymem_leak.c: NAN...

Checking cymem_leak.c: PYPY_VERSION...

Checking cymem_leak.c: PYREX_WITHOUT_ASSERTIONS...

Checking cymem_leak.c: Py_PYTHON_H...

[cymem_leak.c:621]: (information) --check-library: There is no matching configuration for function Py_XDECREF()

....

....

2) Would you please cythonize this and let us know whether cppcheck catches the memory error with this code or not ?

cdef cymain(): 

  cdef double *x

  x= <double*> malloc(sizeof(double) * 1000)

  #free(x) #Whether commented or uncommented, valgrind output does not change at all

  return 0; 

Thank you.