Using tracemalloc with Cython

[Cristina Bogisich]

Hi all,

When I try to use tracemalloc in compiled Cython code, all the memory allocation statistics get combined into a single <unknown> entry. Is it possible to use tracemalloc with Cython and still see the file names and line numbers like in regular Python?

Thanks!

[D Woods]

Have you tried enabling line tracing (https://cython.readthedocs.io/en/latest/src/tutorial/profiling_tutorial.html#enabling-line-tracing)? I'm not absolutely sure that this will work but that's the most likely option. If that doesn't work then it probably isn't possible.

[Cristina Bogisich]

Yes, I tried that and also added the macro definition described below in that section, but still no file names or line numbers. Do you have any suggestions for other ways I could track memory allocation in Cython? I'm trying to track down a memory leak in a relatively complex Cython program that only appears after a long time (hours of runtime). Also here is the code I'm using for testing tracemalloc:

test.pyx:

# cython: profile=True
# cython: linetrace=True
# distutils: define_macros=CYTHON_TRACE_NOGIL=1

tracemalloc.start()

arr = []
for i in range(10000):
arr.append(i)

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('traceback')
for stat in top_stats:
print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))
for line in stat.traceback.format():
print(line)

setup.py:

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules=cythonize("test.pyx"),
)

Compilation and running:

$ py setup.py build_ext --inplace

$ py

Python 3.10.9 (main, Dec 15 2022, 18:18:30) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import fib

9745 memory blocks: 350.1 KiB

  File "<frozen importlib._bootstrap>", line 241

>>> 

[Peter Schay]

On Thu, Oct 5, 2023 at 12:57 PM 'Cristina Bogisich' via cython-users <cython...@googlegroups.com> wrote:

. Do you have any suggestions for other ways I could track memory allocation in Cython?

Hi,

I’ve used guppy in the past.   It has a high learning curve; but it’s fun and very useful after a bit of experimenting.

Regards,

Pete

[Peter Schay]

Pardon me for replying to my own email —

I should also have mentioned that it’s possible to use valgrind.  IIRC it’s necessary to build cpython from source in  a specific way first.

[Cristina Bogisich]

Thank you. I can't add external Python libraries to this project so I won't be able to use guppy, but I will check out valgrind!

[Peter Schay]

I think Valgrind can only help with a "true" memory leak, at the C level.   Which is certainly possible when dealing with lots of cython and C code; however higher level leaks in Python are more common.

If your leak is higher up in the python runtime, such as a dictionary that keeps growing, then guppy is perfect, so just use it for a day and don't tell anyone :-)

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/52e16bf7-9d86-4d46-89c2-db005b88495en%40googlegroups.com.

[Prakhar Goel]

I've had some luck just messing around with the Python gc module as well. It takes a little more work but it (combined with sys.getsize) can be surprisingly effective.

--
________________________
Warm Regards
Prakhar Goel

To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/CALKyFjYk2WaqEWG2J3Un_dLeV%3DBPAgmtG7QeOBJg9tBNZ1LsJw%40mail.gmail.com.