Using gperftools, not able to see all nodes in the call graph of the lower dependencies - google-perftools

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Using gperftools, not able to see all nodes in the call graph of the lower dependencies
Received-SPF: pass (google.com: domain of dvsqu...@gmail.com designates 209.85.214.49 as permitted sender) client-ip=209.85.214.49;
MIME-Version: 1.0
In-Reply-To: <CALj-ATKD9avPLJ3MAP=bBLjvx-kAbR8jVwm4EXUYLrR3C+X...@mail.gmail.com>
References: <55f6c261-7e8b-4c5c-a372-d410c754dea7@googlegroups.com>
	<CALj-AT+OYBRM0aFy=WsXpF0E1ER0_7W8RQ5-yx+Li1YTCV1...@mail.gmail.com>
	<27877617-e240-4171-8fe4-aed484b97deb@googlegroups.com>
	<CALj-AT+SJwAQ-3FaKra-hMd41UEyB+Rt_cLBU_pyP1yzNci...@mail.gmail.com>
	<CAFROQOP+ftNjSX=BJ4R4CLFcN2Sj+mh0umu==sWJCHxfYF-...@mail.gmail.com>
	<CALj-ATKD9avPLJ3MAP=bBLjvx-kAbR8jVwm4EXUYLrR3C+X...@mail.gmail.com>
Date: Tue, 6 Nov 2012 23:56:15 +0530
Message-ID: <CAFROQOPQ-6vyQVyTW7Cidb79KyJRdBKdXpKaEmqV9vC=PMV...@mail.gmail.com>
Subject: Re: Using gperftools, not able to see all nodes in the call graph of
 the lower dependencies
From: Deepak Vijayvergiy <dvsqu...@gmail.com>
To: google-perftools@googlegroups.com
Content-Type: multipart/alternative; boundary=0015175881d6cc72d304cdd7bc51

--0015175881d6cc72d304cdd7bc51
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

yes, i have already doing CPUPROFILE_FREQUENCY =3D 10000.
And now, I think I have been getting good results, when running my code
with good amount of inputs in another way.

Thanks for all the help, responses David.

Regards
Deepak

On Tue, Nov 6, 2012 at 10:40 PM, David Chappelle <chapp...@gmail.com> wrote=
:

> I haven't spent a whole lot of time learning all of the ins and outs of
> the cpu profiler but you might need to be a bit careful with CPUPROFILE_F=
REQUENCY.
> For example, setting it too high/low.
>
> -Dave
>
> On Tue, Nov 6, 2012 at 12:01 PM, Deepak Vijayvergiy <dvsqu...@gmail.com>w=
rote:
>
>> Hi David,
>> To add one more thing, I found out that even if I was using latest
>> gperftools, but it didn't have patch for CPUPROFILESIG feature, which I =
had
>> patched on my gperftool codebase and re-built again.
>> Now, I am able to start/stop profiling by giving signal 12, and it is
>> kind of working. But I am getting a little less number of samples, by wh=
ich
>> I am not able to exactly pin-point a few functions which could be taking
>> the most of time.
>>
>> Thanks
>> Regards
>> Deepak
>>
>>
>> On Tue, Nov 6, 2012 at 7:07 PM, David Chappelle <chapp...@gmail.com>wrot=
e:
>>
>>> Ok, the forked process will get profiled via the cpu profiler and shoul=
d
>>> have a profile suffixed with its pid. The profile for your main process
>>> should not have a pid suffix. Although, it kind of sounds like your mai=
n
>>> process doesn't actually have the cpu profiler linked in. I would try t=
he
>>> LD_PRELOAD approach to make sure that the profiler is in fact being
>>> included. Also, you could have your process hang early on using a 'stat=
ic
>>> bool waitforgdb=3Dtrue; while (waitforgdb) {}' type of code block, then
>>> attach with gdb and set a break point in the cpu profilers destructor. =
From
>>> there you should be able to determine why the profile isn't being gener=
ated.
>>>
>>>
>>> On Tue, Nov 6, 2012 at 1:47 AM, Deepak Vijayvergiy <dvsqu...@gmail.com>=
wrote:
>>>
>>>> Hi,
>>>> Yes, our deamon process is forking one child process.
>>>> Also, at the time of posting this query, we were using google perf
>>>> tools 1.7.x
>>>> Now, we are using gperftools so as to make use of the signal feature
>>>> coming in 2.1
>>>> When we tried CPUPROFILESIGNAL=3D12, to start/stop profiling, there al=
so
>>>> we couldn't even collect the profile file itself.
>>>>
>>>>
>>>> Regards
>>>> Deepak
>>>>
>>>> On Tuesday, 6 November 2012 05:38:31 UTC+5:30, chappelle wrote:
>>>>
>>>>> Is your daemon process forking child processes?
>>>>> Is your deamon or utility command using signal handling? Possibly
>>>>> interfering with the cpu profilers signal handling?
>>>>> You can try running with PERFTOOLS_VERBOSE=3D99 to see if you get any
>>>>> additional logging info.
>>>>>
>>>>>
>>>>> On Mon, Nov 5, 2012 at 3:29 AM, Deepak Vijayvergiy <dvsq...@gmail.com=
>wrote:
>>>>>
>>>>>>  Hi,
>>>>>> I have been trying to use google perftools to profile my product, bu=
t
>>>>>> facing a few issues.
>>>>>> We have a dependency like this -
>>>>>> main_server (deamon running which will take up all the DNS queries
>>>>>> coming in)
>>>>>> Then it will make the threads to take care of dns resolutions, and
>>>>>> that library is thread_lib.
>>>>>> This thread_lib will call the our_dns_lib library to lookup and
>>>>>> resolve the query and respond.
>>>>>>
>>>>>> The dep:
>>>>>> main_server - > thread_lib -> our_dns_lib -> =85 -> =85.
>>>>>>
>>>>>> I have all the above linked with -lprofiler at built-time.
>>>>>>
>>>>>> Now, the way I have tried like this -
>>>>>> (1) Started the deamon with the env CPUPROFILE=3D<prof file>
>>>>>> (2) Now, send the queries through the other terminal in the same
>>>>>> environment.
>>>>>> (3) As soon as deamon receives the first query, it process and
>>>>>> responds and also this line generated -
>>>>>> PROFILE: interrupts/evictions/bytes =3D 10/0/380
>>>>>> (4) Ideally, it shouldn't have generated this line and the profile
>>>>>> file untill we are done with sending all the queries and stop the de=
amon,
>>>>>> am I correct?
>>>>>> (5) I tried to read the profile file generated using the pprof
>>>>>> --text, but it only contained information about the main_server (dea=
mon
>>>>>> stuff). It didn't contain much nodes about the other dependent libra=
ries.
>>>>>> Please point out, if something is missing here?
>>>>>>
>>>>>> After that, as another experiment, I tried as below -
>>>>>> (1) Since, I was very much interested in one of the dependent
>>>>>> libraries nodes, so I added ProfilerStart() and stop() function with=
 the
>>>>>> profile file name as argument in the lookup() function of the depend=
ent
>>>>>> library (our_dns_lib). Compiled it and loaded the new libraries in t=
he
>>>>>> system.
>>>>>> (2) Now, I ran the deamon without using the env CPUPROFILE, and it
>>>>>> started.
>>>>>> (3) Now, after sending the query/packet, I get the response, and I
>>>>>> have checked that it is going to the our_dns_lib library (as we chec=
ked in
>>>>>> ur logs), but profiling is not kicking in somehow.
>>>>>> (4) But, to simulate the lookup() function in our our_dns_lib
>>>>>> library, we have a utility command to test the lookup function, if I=
 use
>>>>>> that, I see profiling kicking in, but I get this -
>>>>>> PROFILE: interrupts/evictions/bytes =3D 0/0/32
>>>>>> Later, I changed the env CPUPROFILE_FREQUENCY=3D10000, and then ran =
the
>>>>>> utility command, it showed something -
>>>>>> PROFILE: interrupts/evictions/bytes =3D 1/0/64
>>>>>>
>>>>>> But, when I tried reading this using pprof, I only get 1 function
>>>>>> (different each time).
>>>>>> So, finally no luck in profiling my code yet.
>>>>>>
>>>>>> Please let me know if I am missing something significant in the
>>>>>> process.
>>>>>> And also, I need to take care of some thing in the process?
>>>>>>
>>>>>> Thanks
>>>>>> Regards
>>>>>>
>>>>>> Deepak
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "google-perftools" group.
>>>>>> To view this discussion on the web visit https://groups.google.com/d=
/
>>>>>> **msg/google-perftools/-/**1mZ2UchSnTkJ<https://groups.google.com/d/=
msg/google-perftools/-/1mZ2UchSnTkJ>
>>>>>> .
>>>>>>  To post to this group, send email to google-p...@googlegroups.**com=
.
>>>>>> To unsubscribe from this group, send email to google-perftoo...@**
>>>>>> googlegroups.com.
>>>>>>
>>>>>> For more options, visit this group at http://groups.google.com/**
>>>>>> group/google-perftools?hl=3Den<http://groups.google.com/group/google=
-perftools?hl=3Den>
>>>>>> .
>>>>>>
>>>>>
>>>>>  --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "google-perftools" group.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msg/google-perftools/-/H2fFfs6ozXEJ.
>>>>
>>>> To post to this group, send email to google-perftools@googlegroups.com=
.
>>>> To unsubscribe from this group, send email to
>>>> google-perftools+unsubscribe@googlegroups.com.
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/google-perftools?hl=3Den.
>>>>
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "google-perftools" group.
>>> To post to this group, send email to google-perftools@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-perftools+unsubscribe@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-perftools?hl=3Den.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google Group=
s
>> "google-perftools" group.
>> To post to this group, send email to google-perftools@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-perftools+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-perftools?hl=3Den.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "google-perftools" group.
> To post to this group, send email to google-perftools@googlegroups.com.
> To unsubscribe from this group, send email to
> google-perftools+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-perftools?hl=3Den.
>

--0015175881d6cc72d304cdd7bc51
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

yes, i have already doing CPUPROFILE_FREQUENCY =3D 10000.<br>And now, I thi=
nk I have been getting good results, when running my code with good amount =
of inputs in another way.<br><br>Thanks for all the help, responses David.<=
br>
<br>Regards<br>Deepak<br><br><div class=3D"gmail_quote">On Tue, Nov 6, 2012=
 at 10:40 PM, David Chappelle <span dir=3D"ltr">&lt;<a href=3D"mailto:chapp=
e...@gmail.com" target=3D"_blank">chapp...@gmail.com</a>&gt;</span> wrote:<b=
r><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex">
I haven&#39;t spent a whole lot of time learning all of the ins and outs of=
 the cpu profiler but you might need to be a bit careful with=A0<span style=
=3D"color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">CPUPRO=
FILE_FREQUENCY. For example, setting it too high/low.</span><div>

<font color=3D"#222222" face=3D"arial, sans-serif"><br></font></div><div><f=
ont color=3D"#222222" face=3D"arial, sans-serif">-Dave<br></font><div><div =
class=3D"h5"><br><div class=3D"gmail_quote">On Tue, Nov 6, 2012 at 12:01 PM=
, Deepak Vijayvergiy <span dir=3D"ltr">&lt;<a href=3D"mailto:dvsquare@gmail=
.com" target=3D"_blank">dvsqu...@gmail.com</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi David,<br>To add one more thing, I found =
out that even if I was using latest gperftools, but it didn&#39;t have patc=
h for CPUPROFILESIG feature, which I had patched on my gperftool codebase a=
nd re-built again.<br>

Now, I am able to start/stop profiling by giving signal 12, and it is kind =
of working. But I am getting a little less number of samples, by which I am=
 not able to exactly pin-point a few functions which could be taking the mo=
st of time.<br>


<br>Thanks<br>Regards<span><font color=3D"#888888"><br>Deepak</font></span>=
<div><div><br><br><div class=3D"gmail_quote">On Tue, Nov 6, 2012 at 7:07 PM=
, David Chappelle <span dir=3D"ltr">&lt;<a href=3D"mailto:chapp...@gmail.co=
m" target=3D"_blank">chapp...@gmail.com</a>&gt;</span> wrote:<br>


<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Ok, the forked process will get profiled via=
 the cpu profiler and should have a profile suffixed with its pid. The prof=
ile for your main process should not have a pid suffix. Although, it kind o=
f sounds like your main process doesn&#39;t actually have the cpu profiler =
linked in. I would try the LD_PRELOAD approach to make sure that the profil=
er is in fact being included. Also, you could have your process hang early =
on using a &#39;static bool waitforgdb=3Dtrue; while (waitforgdb) {}&#39; t=
ype of code block, then attach with gdb and set a break point in the cpu pr=
ofilers destructor. From there you should be able to determine why the prof=
ile isn&#39;t being generated.<div>


<div><br>
<br><div class=3D"gmail_quote">On Tue, Nov 6, 2012 at 1:47 AM, Deepak Vijay=
vergiy <span dir=3D"ltr">&lt;<a href=3D"mailto:dvsqu...@gmail.com" target=
=3D"_blank">dvsqu...@gmail.com</a>&gt;</span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex">



Hi,<br>Yes, our deamon process is forking one child process.<br>Also, at th=
e time of posting this query, we were using google perf tools 1.7.x<br>Now,=
 we are using gperftools so as to make use of the signal feature coming in =
2.1<br>



When we tried CPUPROFILESIGNAL=3D12, to start/stop profiling, there also we=
 couldn&#39;t even collect the profile file itself.<div><br><br>Regards<br>=
Deepak<br><br>On Tuesday, 6 November 2012 05:38:31 UTC+5:30, chappelle  wro=
te:</div>



<blockquote class=3D"gmail_quote" style=3D"margin:0;margin-left:0.8ex;borde=
r-left:1px #ccc solid;padding-left:1ex"><div>Is your daemon process forking=
 child processes?<br>Is your deamon or utility command using signal handlin=
g? Possibly interfering with the cpu profilers signal handling?<br>



</div><div><div>You can try running with=A0PERFTOOLS_VERBOSE=3D99 to see if=
 you get any additional logging info.</div><div>
<br><div><br><div class=3D"gmail_quote"><div><div>On Mon, Nov 5, 2012 at 3:=
29 AM, Deepak Vijayvergiy <span dir=3D"ltr">&lt;<a>dvsq...@gmail.com</a>&gt=
;</span> wrote:<br></div></div><blockquote class=3D"gmail_quote" style=3D"m=
argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">



<div><div>
Hi,<br>I have been trying to use google perftools to profile my product, bu=
t facing a few issues.<br>We have a dependency like this - <br>main_server =
(deamon running which will take up all the DNS queries coming in)<br>Then i=
t will make the threads to take care of dns resolutions, and that library i=
s thread_lib.<br>




This thread_lib will call the our_dns_lib library to lookup and resolve the=
 query and respond.<br><br>The dep:<br>main_server - &gt; thread_lib -&gt; =
our_dns_lib -&gt; =85 -&gt; =85.<br><br>I have all the above linked with -l=
profiler at built-time.<br>




<br>Now, the way I have tried like this -<br>(1) Started the deamon with th=
e env CPUPROFILE=3D&lt;prof file&gt;<br>(2) Now, send the queries through t=
he other terminal in the same environment.<br>(3) As soon as deamon receive=
s the first query, it process and responds and also this line generated -<b=
r>




PROFILE: interrupts/evictions/bytes =3D 10/0/380<br>(4) Ideally, it shouldn=
&#39;t have generated this line and the profile file untill we are done wit=
h sending all the queries and stop the deamon, am I correct?<br>(5) I tried=
 to read the profile file generated using the pprof --text, but it only con=
tained information about the main_server (deamon stuff). It didn&#39;t cont=
ain much nodes about the other dependent libraries.<br>




Please point out, if something is missing here?<br><br>After that, as anoth=
er experiment, I tried as below - <br>(1) Since, I was very much interested=
 in one of the dependent libraries nodes, so I added ProfilerStart() and st=
op() function with the profile file name as argument in the lookup() functi=
on of the dependent library (our_dns_lib). Compiled it and loaded the new l=
ibraries in the system.<br>




(2) Now, I ran the deamon without using the env CPUPROFILE, and it started.=
<br>(3) Now, after sending the query/packet, I get the response, and I have=
 checked that it is going to the our_dns_lib library (as we checked in ur l=
ogs), but profiling is not kicking in somehow.<br>




(4) But, to simulate the lookup() function in our our_dns_lib library, we h=
ave a utility command to test the lookup function, if I use that, I see pro=
filing kicking in, but I get this -<br>PROFILE: interrupts/evictions/bytes =
=3D 0/0/32<br>




Later, I changed the env CPUPROFILE_FREQUENCY=3D10000, and then ran the uti=
lity command, it showed something -<br>PROFILE: interrupts/evictions/bytes =
=3D 1/0/64<br><br>But, when I tried reading this using pprof, I only get 1 =
function (different each time).<br>




So, finally no luck in profiling my code yet.<br><br>Please let me know if =
I am missing something significant in the process.<br>And also, I need to t=
ake care of some thing in the process?<br><br>Thanks<br>Regards</div></div>



<span><font color=3D"#888888"><div><div><br>
Deepak<br><div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;google-perftools&quot; group.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/d/msg/google-perftools/-/1mZ2UchSnTkJ" target=3D"_blank">https://groups.=
google.com/d/<u></u>msg/google-perftools/-/<u></u>1mZ2UchSnTkJ</a>.<br></di=
v>



</div>=20
To post to this group, send email to <a>google-p...@googlegroups.<u></u>com=
</a>.<br>
To unsubscribe from this group, send email to <a>google-perftoo...@<u></u>g=
ooglegroups.com</a>.<div><br>

For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/google-perftools?hl=3Den" target=3D"_blank">http://groups.google.com/<=
u></u>group/google-perftools?hl=3Den</a>.<br>


</div></font></span></blockquote></div><br></div></div></div>
</blockquote><div><div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;google-perftools&quot; group.<br></div>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/d/msg/google-perftools/-/H2fFfs6ozXEJ" target=3D"_blank">https://groups.=
google.com/d/msg/google-perftools/-/H2fFfs6ozXEJ</a>.<div><div>
<br>=20
To post to this group, send email to <a href=3D"mailto:google-perftools@goo=
glegroups.com" target=3D"_blank">google-perftools@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href=3D"mailto:google-perf=
tools%2Bunsubscribe@googlegroups.com" target=3D"_blank">google-perftools+un=
subscribe@googlegroups.com</a>.<br>

For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/google-perftools?hl=3Den" target=3D"_blank">http://groups.google.com/g=
roup/google-perftools?hl=3Den</a>.<br>


</div></div></blockquote></div><br>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;google-perftools&quot; group.<br>
To post to this group, send email to <a href=3D"mailto:google-perftools@goo=
glegroups.com" target=3D"_blank">google-perftools@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href=3D"mailto:google-perf=
tools%2Bunsubscribe@googlegroups.com" target=3D"_blank">google-perftools+un=
subscribe@googlegroups.com</a>.<br>

For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/google-perftools?hl=3Den" target=3D"_blank">http://groups.google.com/g=
roup/google-perftools?hl=3Den</a>.<br>


</div></div></blockquote></div><br><div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;google-perftools&quot; group.<br>
To post to this group, send email to <a href=3D"mailto:google-perftools@goo=
glegroups.com" target=3D"_blank">google-perftools@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href=3D"mailto:google-perf=
tools%2Bunsubscribe@googlegroups.com" target=3D"_blank">google-perftools+un=
subscribe@googlegroups.com</a>.<br>

For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/google-perftools?hl=3Den" target=3D"_blank">http://groups.google.com/g=
roup/google-perftools?hl=3Den</a>.<br>


</div></div></blockquote></div><br></div></div></div><div class=3D"HOEnZb">=
<div class=3D"h5">

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;google-perftools&quot; group.<br>
To post to this group, send email to <a href=3D"mailto:google-perftools@goo=
glegroups.com" target=3D"_blank">google-perftools@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href=3D"mailto:google-perf=
tools%2Bunsubscribe@googlegroups.com" target=3D"_blank">google-perftools+un=
subscribe@googlegroups.com</a>.<br>

For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/google-perftools?hl=3Den" target=3D"_blank">http://groups.google.com/g=
roup/google-perftools?hl=3Den</a>.<br>


</div></div></blockquote></div><br><div style id=3D"__af745f8f43-e961-4b88-=
8424-80b67790c964__"></div>

--0015175881d6cc72d304cdd7bc51--
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
