How does pprof work under the hood?

[ben1s...@gmail.com]

I am looking for a simple explanation of how the pprof library works. So far, this is what I have learned:

• It runs a stop-the-world sampler. In other words, it periodically stops the program being profiled to collect information.
• It uses gperftools underneath.

Besides a general overview, here are some specific questions I would like answered:

• Is pprof an "event based profiler" or "instrumentation profiler". From what I understand, these profilers modify the way a program runs and collect samples via those modifications
• At what 'level' in the OS does pprof profile? Does it profile the kernal like SystemTap or perf?
• Is pprof safe to run on a high-traffic production server?

I am asking this question to reason about the overhead introduced by using pprof on a Go server.

[Ian Lance Taylor]

On Thu, Jun 18, 2015 at 12:25 AM, <ben1s...@gmail.com> wrote:
>
> I am looking for a simple explanation of how the pprof library works.

I assume you are talking about the runtime/pprof package, rather than,
say, the cmd/pprof tool

> So far, this is what I have learned:
>
> It runs a stop-the-world sampler. In other words, it periodically stops the
> program being profiled to collect information.
> It uses gperftools underneath.

Hmmm, maybe you aren't talking about runtime/pprof after all.
Actually I'm not sure what you are talking about.

> Is pprof an "event based profiler" or "instrumentation profiler". From what
> I understand, these profilers modify the way a program runs and collect
> samples via those modifications

The runtime/pprof package is an event based profiler.

> At what 'level' in the OS does pprof profile? Does it profile the kernal
> like SystemTap or perf?

For runtime/pprof, no.

> Is pprof safe to run on a high-traffic production server?

For runtime/pprof, yes.

Ian

[ben1s...@gmail.com]

Thank you for the answer! I am talking about the http/pprof package (which I believe wraps the runtime/pprof package). 

• Is it true that pprof is a stop-the-world sampler? Does it periodically stop the program being profiled to collect information?
• Is pprof a statistical/stochastic profile? Is it also an event based profiler? 
• I know it doesn't run at the kernal level, but where in the OS does pprof profile?

Ben

[Ian Lance Taylor]

On Thu, Jun 18, 2015 at 10:38 AM, <ben1s...@gmail.com> wrote:

> Thank you for the answer! I am talking about the http/pprof package (which I
> believe wraps the runtime/pprof package).
>
> Is it true that pprof is a stop-the-world sampler? Does it periodically stop
> the program being profiled to collect information?

No.

> Is pprof a statistical/stochastic profile?

Yes.

> Is it also an event based profiler?

Yes.

> I know it doesn't run at the kernal level, but where in the OS does pprof
> profile?

As you say, the net/http/pprof package is basically a wrapper around
the runtime/pprof package.

For CPU profiling, runtime/pprof works by periodically interrupting
the program. On Unix-like systems, this is done using setitimer to
send a periodic SIGPROF signal. When this signal arrive, the
goroutine that receives it stores a stack trace. This very briefly
stops the goroutine being profiled, but it doesn't affect the rest of
the program.

For heap profiling, the memory allocator tracks the number of
allocations it has done, and periodically records a stack trace.

Ian