Hello Folks,
I'm cpu-profiling a multi-threaded application on a linux 32 bit
machine.

The profiler captures about 1000 samples, but no two nodes on the
graph are connected.
In the textual version, "Percentage of profiling samples in this
function" & "Percentage of profiling samples in this function and its
callees" are same on every line.

I'm using ProfilerStart() and ProfilerStop() methods to trigger the
start & stop of profiling. The binary is linked with libprofiler.so.
It is definitely profiling all my threads.

The CPUPROFILE_FREQUENCY is set to 4000.

Any guess why its not able to link the method calls? Is that something
to do with stack unwind issues?

Appreciate your reply.

Thanks
Raju

It could be.  But on 32 bit machines, you'd expect stack unwinding to
work ok.

As an experiment, try commenting out the ProfilerStart/Stop calls, and
then running your binary with
   env CPUPROFILE=/var/tmp/whatever ./your_binary

Does that graph look any better?

craig

Setting the CPUPROFILE didn't show any difference.

I had perftools 0.99 profiling successfully on few other multi-
threaded applications. So, I tried 0.99 build again on this one. And
to my surprise, the graph was perfect.

Any guess why an older build is good? Any specific configure/make
options that I should pay attention to in the new releases?

Thank you for your time

cheers
Raju

On Jun 22, 7:23 pm, Craig Silverstein <csilv...@google.com> wrote:

Weird.  Can you maybe include the generated profile files in these two
cases (0.99 and 1.3)?  I wonder if the list of symbols is different,
or they're just being resolved differently, or what.

craig

Hi,

I have the same problem, but with both versions (1.3 and 0.99.1). I
get a long list of nodes, which are not connected. Single-thread
application with dynamic linking. I start pprof like this:

libtool --mode=execute pprof prog ./prog.prof

Any hints?

Arne

On Jun 29, 10:17 pm, Craig Silverstein <csilv...@google.com> wrote:

Hmm, it could be any number of things.  What architecture are you on?
It may be you're on x86_64 and have compiled perftools to use frame
pointers, but the dynamic libraries don't have frame pointers.  What
do your profile files look like?

craig

Hi,

first, thanks for answering!

On Di, 2009-07-07 at 10:07 -0700, Craig Silverstein wrote:

Ah, totally forgot to include this information *blush*. I'm on x86,
Gentoo Linux (2.6.29-gentoo-r5).

No, I compiled it without any special flags -- just the standard set
used by gentoo:

./configure --prefix=/usr --build=i686-pc-linux-gnu  
        --host=i686-pc-linux-gnu --mandir=/usr/share/man
        --infodir=/usr/share/info --datadir=/usr/share
        --sysconfdir=/etc --localstatedir=/var/lib

Well, I don't really know what you mean by that. The output is always
the same, a list of nodes:

Total: 126907 samples
   36502  28.8%  28.8%    36502  28.8% monstartup
   13256  10.4%  39.2%    13256  10.4% mcount
    6564   5.2%  44.4%     6564   5.2% swarmanoid::C3DMath::intersects
    5043   4.0%  48.4%     5043   4.0% __i686.get_pc_thunk.bx
    4964   3.9%  52.3%     4964   3.9% h_errno
    4868   3.8%  56.1%     4868   3.8% _init
    3990   3.1%  59.2%     3990   3.1% _nv000125gl
    3660   2.9%  62.1%     3660   2.9% swarmanoid::CVector3::CVector3
     ..     ..    ..

I sent the profile in case you want to have a look at it.

Regards,
Arne

Thank you for sending the profile file my way.  I looked at it, and
indeed, it shows a bunch (thousands) of unconnected nodes.  It's
almost like you weren't able to propertly get a stack trace.

What happens when you try to create a heap profile?  Does that work
any better?  If so, then the problem is with the cpu-profiler specific
code, perhaps getting the PC (did you see any errors regarding that
when you built the perftools package)?  If heap profile doesn't work
either, then the problem is more likely to be with getstracktrace or
similar.

Have you tried running the unittests that come with the package?  Do
they all pass?

craig

Hello,

No, worse actually. I get the following error:
"Hooked allocator frame not found, returning empty trace"

This seems to be the case. Might this be a problem with my global build
options on gentoo? I've got:

        CFLAGS="-march=prescott -O2 -pipe -fomit-frame-pointer"
        CXXFLAGS="${CFLAGS}"
        MAKEOPTS="-j5"
        CHOST="i686-pc-linux-gnu"

I rebuild 1.3, no errors at all. I ran the following unittests:

        heap-checker-death_unittest.sh
        heap-checker_unittest.sh
        heap-profiler_unittest.sh
        maybe_threads_unittest.sh
        profiler_unittest.sh

All of them passed. Might this have something todo with the dynamic
library loading we employ in our program? It's a very modular piece of
software, and we load almost all functionality as dynamic libraries
configured by an xml.

Additionally, the two largest nodes in the output are "monstartup" and
"mcount". These are usually used with gprof profiling support, right? I
disabled all -pg flags, they still show up. Does this have anything to
do with it or are you using the same methods for profiling?

Sorry, I really have no clue about the inner workings of this stuff.

Cheers,
Arne

That would definitely do it!  Without frame pointers, you can't get
any useful stack trace information.  If you can rebuild with frame
pointers, you should be in much better shape for these tools.

craig

That did the trick, thanks! In the end, I just used my other Linux box
(Ubuntu), and it's fine.

Just a note concerning the .debs: I had to manually install graphviz,
apparently it's not listed in the debs dependencies.

Thanks
arne

On Fr, 2009-07-10 at 11:42 -0700, Craig Silverstein wrote: