Memory profiling a Pyramid app

[Vlad K.]

How? Which tools?

I'm trying to use heapy but I'm fighting both with lack of proper
documentation, and with the fact that I need to profile a wsgi app run
behind uwsgi. Even if I run with pserve (1.3) I have no idea how to use
heapy's interactive features.

What else would you suggest to memory profile an app? Preferably with
simple but effective statistics of heapy (count, % count, size, % size
and cumulative)?



Thanks.

--

.oO V Oo.

[Marius Gedminas]

I'd be inclined to try out Pyrasite (http://pyrasite.com/). It attaches
to a running process with ptrace and lets you do stuff. The GUI has a
tab for doing memory profiling with Meliae (https://launchpad.net/meliae),
which is similar to heapy, AFAIU.

You can also use RunSnakeRun (http://pypi.python.org/pypi/RunSnakeRun) to
visualize the JSON of allocated object stats you get from Meliae.

Dozer (http://pypi.python.org/pypi/Dozer) is a WSGI middleware memory
profiler that might be useful. I've never used it. AFAIU it gives you
a web page with a bunch of small graphs showing allocated object counts
by type.

I've also experimented with sticking a pdb.set_trace() somewhere in the
code, looking at

(pdb) gc.collect()
(pdb) for n in range(20): print random.choice(gc.get_objects())

hoping that statistics will work in my favour (most frequently allocated
objects would be the ones randomly chosen), and then using my own
objgraph (http://pypi.python.org/pypi/objgraph) to trace the reference
chains to see what's holding those objects in memory.

HTH,
Marius Gedminas
--
We have an advanced scalable groupware communication environment (email)
-- Alan Cox

[Jason]

I used objgraph ( http://mg.pov.lt/objgraph/ ) recently to find a memory leak in an application (turns out debugtoolbar was holding references to 100000s of sqlalchemy objects). I didn't use all the features but I saw a count of the different type of objects and then you can also graph the references to a specific instance as well. You can also set a baseline and then see how many items of each type are created after that baseline.

-- Jason

[Vlad K.]

Thanks for all your replies.


I've also tried pympler and looks like nice alternative to heapy().


.oO V Oo.