pprof samples

[asa...@arista.com]

Hi,

I am trying CPU profiling a program that runs for an hour. However when I start pprof it says Total samples are only for 1.32 min. Why are the samples not collected for the whole duration of the program ? Is there a missing setting I need to enable ?

Duration: 1.14hrs, Total samples = 1.32mins ( 1.94%)

Thanks.

[Diego Medina]

Hi,

This doesn't answer your question but usually when I need to profile our app, I first use expvar to report different internal stats like what the app is doing (how many records is it walking from the db, any batch process, etc), send those to datadog (any monitoring works fine) and then I know "which code" is the one that is causing high CPU/memory.

Then it's a matter to writing simple go benchmark code (sometimes even talking to the db, I don't bother with mocking) and use pprof on the benchmark code

Hope that helps.

Diego

[David Finkel]

On Mon, Apr 20, 2020 at 8:35 PM asaxena via golang-nuts <golan...@googlegroups.com> wrote:

Hi,

I am trying CPU profiling a program that runs for an hour. However when I start pprof it says Total samples are only for 1.32 min. Why are the samples not collected for the whole duration of the program ? Is there a missing setting I need to enable ?

It sounds like it was collected fro the duration of the program (see below), no additional configuration needed. 

Duration: 1.14hrs, Total samples = 1.32mins ( 1.94%) 

This generally means that your process only used (roughly) 1.32 minutes of CPU-time over the 1.14hrs of the profile. Equivalent to an average CPU-usage of the 1.94% it listed.

My usual inference from this kind of profile is that this program is not CPU-bound and instead I/O or network-bound.

CPU-profiling is only useful for debugging cpu-bound operations, it doesn't tell you where operations were blocked, contending, etc.

Diego's suggestion of instrumenting your code with some sort of metrics library (I'm a fan of opencensus's stats + prometheus) is rather useful for tracking aggregate stats.

In general, though, if you have a specific request/process you want to find the bottleneck in, it's hard to get away from tracing. I've used opentracing with Zipkin, but most recently I use opencensus tracing with stackdriver. For a single process, though I just use opencensus's zpages to get a view of the current trace-spans on that process.

Thanks.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a13db751-0f65-4b0b-aaf3-f39bc29e21cf%40googlegroups.com.

[Robert Engels]

pprof is not only cpu profiling - it is the same tracing infrastructure used in the go analyzer. 

On Apr 22, 2020, at 2:28 PM, David Finkel <david....@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CANrC0BizHrEy73JWJZHPrVx2Wb91z0vPj87HgLcg6nu1ydg0Mg%40mail.gmail.com.

[asa...@arista.com]

The program is indeed IO and network bound so it does seem the CPU usage is minimal. 

Thanks for all the pointers for tracing and analyzing !!

On Wednesday, April 22, 2020 at 2:45:57 PM UTC-7, Robert Engels wrote:

pprof is not only cpu profiling - it is the same tracing infrastructure used in the go analyzer. 

On Apr 22, 2020, at 2:28 PM, David Finkel <david...@gmail.com> wrote:



On Mon, Apr 20, 2020 at 8:35 PM asaxena via golang-nuts <golan...@googlegroups.com> wrote:

Hi,

I am trying CPU profiling a program that runs for an hour. However when I start pprof it says Total samples are only for 1.32 min. Why are the samples not collected for the whole duration of the program ? Is there a missing setting I need to enable ?

It sounds like it was collected fro the duration of the program (see below), no additional configuration needed. 

Duration: 1.14hrs, Total samples = 1.32mins ( 1.94%) 

This generally means that your process only used (roughly) 1.32 minutes of CPU-time over the 1.14hrs of the profile. Equivalent to an average CPU-usage of the 1.94% it listed.

My usual inference from this kind of profile is that this program is not CPU-bound and instead I/O or network-bound.

CPU-profiling is only useful for debugging cpu-bound operations, it doesn't tell you where operations were blocked, contending, etc.

Diego's suggestion of instrumenting your code with some sort of metrics library (I'm a fan of opencensus's stats + prometheus) is rather useful for tracking aggregate stats.

In general, though, if you have a specific request/process you want to find the bottleneck in, it's hard to get away from tracing. I've used opentracing with Zipkin, but most recently I use opencensus tracing with stackdriver. For a single process, though I just use opencensus's zpages to get a view of the current trace-spans on that process.

Thanks.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a13db751-0f65-4b0b-aaf3-f39bc29e21cf%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.