Doubts regarding implementation of runtime profiler

[Nidhi Agrawal]

Hi,

Golang runtime profiler (for cpu & trace) gives us an option to decide the time of profiling. It then sets the rate of profiling at start of the call and reset the rate at the end of the call. So it goes like this

CPUProfiling() {
   StartCPUProfile() // internally calls SetCPUProfileRate(100)
   sleep(seconds)
   StopCPUProfile () // internally calls SetCPUProfileRate(0)
}

On the other hand for mutex and block profiling, it expects us to set the rate at the start of the application and while calling profiling, it just gathers the information and return.

My doubt is why mutex and block profiling have different behaviour ? Here also we could have set the rate at call start and reset at call end with some sleep time.

Thanks,
Nidhi

[Ian Lance Taylor]

The two kinds of profiling work differently. Memory profiling is
based on sampling allocations as determined by runtime.MemProfileRate.
Adjusting that on the fly won't give you reasonable results, because
of how the profiling tools handle the data. Changing this would
require some mechanism to tell the tools when MemProfileRate changes,
and we don't have a way to do that.

Is this causing a problem?

Ian

[Nidhi Agrawal]

HI Ian,

We don't need to set MemProfileRate. It is about block and mutex profiling where we need to set the rate (SetBlockProfileRate, SetMutexProfileFraction) at application start. Is there any performance impacts of doing that ? 

It is not causing any issue we want to enable block and mutex profiling so i am thinking if we can do something like

  SetBlockProfileRate(1)

  pprof.Lookup("block").WriteTo(f, 0); 

  sleep(20)

  SetBlockProfileRate(0)

just like how cpu and trace profile works. But i am curious why isn't it not already done in the library. What can be the reason behind this?

I am curious why they are different from cpu and trace profiling, trying to understand the inner working of this. It would be really helpful if you can give me some pointers on where i can read about the high level design of the pprof.

Thanks

[Ian Lance Taylor]

On Sat, Oct 19, 2019 at 7:34 PM Nidhi Agrawal <nidhi...@gmail.com> wrote:
>
> We don't need to set MemProfileRate. It is about block and mutex profiling where we need to set the rate (SetBlockProfileRate, SetMutexProfileFraction) at application start. Is there any performance impacts of doing that ?
>
> It is not causing any issue we want to enable block and mutex profiling so i am thinking if we can do something like
> SetBlockProfileRate(1)
> pprof.Lookup("block").WriteTo(f, 0);
> sleep(20)
> SetBlockProfileRate(0)
> just like how cpu and trace profile works. But i am curious why isn't it not already done in the library. What can be the reason behind this?
>
> I am curious why they are different from cpu and trace profiling, trying to understand the inner working of this. It would be really helpful if you can give me some pointers on where i can read about the high level design of the pprof.

Oh, sorry, I misread what you were asking. But now I'm not sure what
you are asking. You said

> On the other hand for mutex and block profiling, it expects us to set the rate at the start of the application and while calling profiling, it just gathers the information and return.

That's how memory profiling works. You don't have to set the mutex
and profiling rates at the start of the application. You can set them
when you want to start profiling, as you suggest. Is there some
documentation that says otherwise?

Sorry, I don't know of any documentation describing the profiling
implementation.

Ian

[Nidhi Agrawal]

https://golang.org/src/runtime/mprof.go, https://golang.org/pkg/net/http/pprof/ here it is mentioned that we need to set SetBlockProfileRate to get enable block profile.

In case of block profiling if I set rate just before profiling and reset it to 0 after profiling i am not getting any data.

I am doing something like:

SetBlockProfileRate(1)

pprof.Lookup("block").WriteTo(f, 0); 

SetBlockProfileRate(0)

[Ian Lance Taylor]

On Sat, Oct 19, 2019 at 10:17 PM Nidhi Agrawal <nidhi...@gmail.com> wrote:
>
>
> https://golang.org/src/runtime/mprof.go, https://golang.org/pkg/net/http/pprof/ here it is mentioned that we need to set SetBlockProfileRate to get enable block profile.
>
> In case of block profiling if I set rate just before profiling and reset it to 0 after profiling i am not getting any data.
>
> I am doing something like:
> SetBlockProfileRate(1)
> pprof.Lookup("block").WriteTo(f, 0);
> SetBlockProfileRate(0)

If that is exactly what you are doing, then that doesn't work for CPU
profiling either. You need to call SetBlockProfileRate, then do some
work, then fetch the profile.

Ian

[Nidhi Agrawal]

Ok, Is it necessary to call SetBlockProfileRate at the start of application?

If not then we can call it just before starting block profile and after block profile interval (eg. 30 seconds) when profiling finishes then we can reset it to 0.

[Ian Lance Taylor]

On Sun, Oct 20, 2019 at 12:23 AM Nidhi Agrawal <nidhi...@gmail.com> wrote:
>
> Ok, Is it necessary to call SetBlockProfileRate at the start of application?
> If not then we can call it just before starting block profile and after block profile interval (eg. 30 seconds) when profiling finishes then we can reset it to 0.

As far as I know you can call SetBlockProfileRate at any time.

Why do you think that it must be called at the start of the application?

[Nidhi Agrawal]

Because it is explicitly written in the documentation here https://golang.org/pkg/net/http/pprof/ that we should set block profile rate at the start of application, unlike CPU profiling where the rate is being set internally before profiling starts. 

As you said we can set the rate any time then similar to cpu profiling, block/mutex profiling can also be implemented like 

go tool pprof http://localhost:6060/debug/pprof/block?seconds=30&rate=1

Where internally we set the rate before profiling starts.

I wanted to understand if there is any reason for such different implementation.

[Ian Lance Taylor]

On Sun, Oct 20, 2019 at 12:36 AM Nidhi Agrawal <nidhi...@gmail.com> wrote:
>
> Because it is explicitly written in the documentation here https://golang.org/pkg/net/http/pprof/ that we should set block profile rate at the start of application, unlike CPU profiling where the rate is being set internally before profiling starts.

I'm likely missing something, but I'm looking at
https://golang.org/pkg/net/http/pprof, and I don't see that. I see
that it says that you have to call runtime.SetBlockProfileRate or
runtime.SetMutexProfileFraction, but I don't see anything that says
that you have to call it at the start of the application.

Ian

[Nidhi Agrawal]

I came to this conclusion because the pprof implemented the cpu with the assumption that the client gives how much time to capture the profiling data. but when it comes to the mutex and block, the pprof didn't implement it to support this but asked the client to call the profile rate before calling the pprof function. This made me question why is there a difference in the profiling technique, is it expected from us to set profile rate at the start of the program, if not then why didn't they implement it similar to the cpu.

[Ian Lance Taylor]

On Sun, Oct 20, 2019 at 12:51 AM Nidhi Agrawal <nidhi...@gmail.com> wrote:
>
> I came to this conclusion because the pprof implemented the cpu with the assumption that the client gives how much time to capture the profiling data. but when it comes to the mutex and block, the pprof didn't implement it to support this but asked the client to call the profile rate before calling the pprof function. This made me question why is there a difference in the profiling technique, is it expected from us to set profile rate at the start of the program, if not then why didn't they implement it similar to the cpu.

I'm sorry, I don't really understand what you mean.

There is really no difference among CPU, block, and mutex profiling
with regard to how they are used. In all cases you start profiling,
do some work, and then fetch the profile. They are all more or less
the same. I don't understand what is leading you to conclude that
they are different.

Heap profiling, on the other hand, really is different, and it is
documented differently in runtime/pprof.

Ian

[Nidhi Agrawal]

Implementation in http/net/pprof library is leading me to conclude that they are different. 

Yes, In runtime/pprof block, mutex, cpu all are implemented in the same way, but http/net/pprof is not using them in the same way.

In CPU profiling rate is set internally, while in block, mutex we are supposed to set the rate.

[Ian Lance Taylor]

On Sun, Oct 20, 2019 at 8:57 PM Nidhi Agrawal <nidhi...@gmail.com> wrote:
>
> Implementation in http/net/pprof library is leading me to conclude that they are different.
> Yes, In runtime/pprof block, mutex, cpu all are implemented in the same way, but http/net/pprof is not using them in the same way.
> In CPU profiling rate is set internally, while in block, mutex we are supposed to set the rate.

Yes, that is true. CPU profiling start with pprof.StartCPUProfile,
while block and mutex profiling start with runtime.SetBlockProfilerate
and runtime.SetMutexProfileFraction. But you don't have to set the
block and mutex rate at the start of the program. You set it when you
want to start profiling.

You can also change the CPU profile rate with
runtime.SetCPUProfileRate, but that is hard to use without
pprof.StartCPUProfile.

[Nidhi Agrawal]

Ok, Also is there any harm if we set block/mutex rate at the start of the program ? Will it cause any memory allocations or any kind of performance impact?

[Ian Lance Taylor]

On Mon, Oct 21, 2019 at 4:38 AM Nidhi Agrawal <nidhi...@gmail.com> wrote:
>
> Ok, Also is there any harm if we set block/mutex rate at the start of the program ? Will it cause any memory allocations or any kind of performance impact?

It will cause some additional memory allocations as the program
records profiling information. But it won't be all that much memory.
The other performance impact should be minimal.

Ian

[Nidhi Agrawal]

Ok, Thanks.