New issue 182 by petef4: pprof results are low compared with top
http://code.google.com/p/google-perftools/issues/detail?id=182

What steps will reproduce the problem?
1. Profile a program (with HEAP_PROFILE_MMAP=true)
2. Compare results with /usr/bin/top
3.

What is the expected output? What do you see instead?
The inuse space reported using pprof is about half of that in the DATA
reported by top.

What version of the product are you using? On what operating system?
1.3 and 1.4.
2.6.24-etchnhalf.1-amd64 #1 SMP Sat Aug 15 20:38:41 UTC 2009 x86_64 GNU/
Linux
export CFLAGS="-m32 -g"
export CXXFLAGS="-m32 -g"
configure --enable-static --enable-frame-pointers --host=i686

Please provide any additional information below.
Top was configured to show extra columns for SWAP, CODE and DATA. Added
columns with f  p r s <enter> and then W to write to .toprc.

Some disparity is expected.
1. TCmalloc does not report usage by stack, initialized data and BSS.
2. TCMalloc has overheads of a few MB
3. TCMalloc reports bytes allocated, top will see these rounded up to
pages
4. TCMalloc does not return freed memory

I'm requesting an enhancement. Add counters for low level calls from
TCMalloc to sbrk, mmap, etc. which can then be reported along with the
program totals.  These new counter results should tally better with top
and provide more confidence that TCMalloc is logging all program usage.

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings

Comment #1 on issue 182 by csilvers: pprof results are low compared with top
http://code.google.com/p/google-perftools/issues/detail?id=182

In our experience, (4) is often the reason for discrepancies with top.
Occassionally, you'll also see discrepancies due to internal fragmentation  
within
tcmalloc.

The way to diagnose these is to emit the output of
MallocExtension::instance()->GetStats() (from malloc_extension.h).  This  
will return
a lot of details about memory use, and should cover all the cases you're  
suggesting
counters for.  In any case, give it a try and see how well it works for you.

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings

Comment #2 on issue 182 by csilvers: pprof results are low compared with top
http://code.google.com/p/google-perftools/issues/detail?id=182

(No comment was entered for this change.)

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings

Craig,

We're getting a lot of situations in production where TCMalloc/pprof appears
to "leak", whereas if we link in another allocator, the memory usage does
not go up.  The "leak" rate appears to be something like 4-5 GB a day on our
server.

Unfortunately, for us, the MallocExtension::getStats() appear to be missing
GBs of data -- a typical situation is that it reports something like 1.9GB
used by heap, another 1.0-1.2GB used by pagecache; meanwhile, 'top' and
other tools report the VIRT and RSS to be >7.5GB.  Similarly,
getHeapSample() analyzed via pprof matches the getStats() and only shows the
1.9GB used.

Some analysis has shown that there are large mmap() blocks found in
/proc/####/maps that are not listed at all on the getStats() information.

I'm not sure if it's the same problem, but we've reproduced this on 4 or 5
binaries in production and the symptoms are always the same, and go away if
we switch to another malloc.

I think this may be the same problem as this ticket -- I can try to provide
more information.

This is the latest perftools with fc4 linux.

- Mark

I would expect GetStats() to show this memory.  GetStats() has weird
behavior that the bigger the buffer you pass in, the more info it
gives you.  Maybe pass in a really big buffer (like 1M) and see if it
gives you more info that might describe where memory is going?

I know it's unlikely, but if you have a simple app that reproduces
this problem (even if it takes a long time to run), I'd love to take a
look at it.

craig