Re: Memory profiling in Devel::NYTProf?
Tim Bunce
On Sun, Jun 21, 2009 at 04:31:10PM +0200, Robin Smidsrød wrote:
>
> I've been discussing memory profiling as of late, and one person (Chris
> Prather) suggested that maybe I should talk to you about including
> something like this in NYTProf before I go about trying to make my own.
>
> How hard would it be to add measurement of memory used by the process in
> NYTProf?
>
> My article,
> http://blog.robin.smidsrod.no/index.php/2009/05/26/memory-footprint-of-popular-cpan-modules,
> shows a simple run of memory profiling for some modules, and I'd like to
> try and extend it to a profiling module.
>
> Is NYTProf made extensible to support this kind of thing, or is memory
> profiling something that wouldn't work with its current architecture?
Fairly simple to do something fairly simple. But tricky to do something
worthwhile.
Copy-on-write, for example, is a significant issue. Then there's the
issue of memory allocated to lexical pads that were only used once, for
example. As is memory allocated to the perl process but not currently
used by it.
Your perlbloat.pl script, for example, doesn't account for modules that
use lots of memory *while loading* but then free much of it (back to
perl's own storage pool). The 'bloat' would be reported too high.
When the next module is loaded it'll use some of that freed memory
so its bloat would be reported too low.
Lots of tricky detail here, in terms of what can be usefully measured
and what it actually means. Lots of thought would also need to be given
to how to report it in a meaningful way.
Having said all that, I am interested in extending NYTProf in this
direction.
> PS: Are you ever on irc.perl.org? If so, under which name?
timbunce, but rarely.
Tim.
Robin Smidsrød
Hash: SHA1
remember where), but yes, it does only measure the total size of memory
allocated to the process. But that in itself is a useful metric. If you
run this script with several modules on the same run you'd get the
_combined_ cost of using them in your program. If you run perlbloat
several times, with only one module as the argument, you get the cost of
that module alone.
Actually there are two metrics that are interesting to measure: The size
of the process (it's called VSS, Virtual Set Size, right?), which only
increases until the process dies, and how much of that memory the perl
interpreter is using of that memory at a given time.
The GTop perl module based on glib from Gnome tries to get this kind of
data in a portable way. That's why I used it in my small script. From
its documentation it seems like it also has the ability to measure
copy-on-write effects with the dirty_size method (AFAIK). If the system
uses a recent linux kernel the perl module Linux::Smaps is also
available to us.
When it comes to measuring how much memory the interpreter is actually
using at a given time (from the allocated) I don't know. Maybe someone
with inside knowledge of perl can shed som light on this? Maybe using a
combination of Devel::Gladiator and Devel::Size can give you an
indication, but I would assume that perl itself keeps a free_process_map
of some sort.
If we go about this in an organized way I think we could make it useful.
* First we need a way to measure the different metrics
* Process size (the one that never decreases)
* Current usage of memory inside above metric
* Possibly shared memory usage aswell?
* memory mapped files (part of the above?)
* Then we need to add the ability to compute differences
* Hook into statement or sub profiler
* Then we need to figure out how to store it in the dumpfile
* And finally we need to figure out how to report it
Anything I've forgotten?
Regards,
Robin Smidsrød
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAko/c4gACgkQHAwEVD/in26MHACgn11CAOM46zPNSX/fjQh1mVpP
ASEAn1V9XdxRinHK+cEnh4kB8vy3QX4M
=RE86
-----END PGP SIGNATURE-----