Any way to reset the memory stats during memory profiling?

[Bruno Albuquerque]

maybe i am just approaching the problem the right way so in case you have some alternative suggestions, I am listening. Here is my problem:

I have this code that loads a lot of data on startup (around 2 Gb in-memory). I was looking to optimize it and profiled the CPU usage. The result clearly showed that the program was generating too much garbage (only GC related functions showed up in the top10 output).

To track that down, i tried to run a memory profile. That did not work as expected because of the amount of memory loaded on startup, the memory profile only showed the code paths related with this initial data loading and nothing else.

So i thought that if there was a way for me to say something like "Data was loaded. Clear memory stats now" i would be able to see what is actually happening.

Does this make any sense at all? If now, any alternatives?

Thanks in advance.

[Péter Szilágyi]

Hi,

  I don't know how to reset the profiler, but just a thought: why don't you start the profiler only after loading the data [1]?

Cheers,

  Peter

[1] http://golang.org/pkg/runtime/#SetBlockProfileRate

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Bruno Albuquerque]

Thanks for the suggestion. This did not seem to wok as expected (at least when profiling using the test facilities). I will try to manually profile it and see if it helps.

-Bruno

[Bruno Albuquerque]

In fact, I just noticed that SetBlockProfileRate is not related to memory profiling at all.

In any case, it seems the -benchmem option for tests can at least give me a rough idea of how big the issue is. And it seems to be big:

BenchmarkCode      20 110851351 ns/op 16353639 B/op  339567 allocs/op

if i am reading it right, a single iteration of the code I am benchmarking resulted in 15 Mb of data being allocated (in 339567 diferent alloc operations). Is this interpretation correct?

-Bruno

[Aaron Blohowiak]

You should do the GOGC=off -memprofilerate=1 trick for a *single* run of your benchmark and then use the resulting memory profile which will show all allocations performed ever (because the heap is never GC'd)

(For this stuff, I especially like "weblist ." in pprof)

[Matthew Zimmerman]

I've never had a problem whittling down go tool pprof to only look at
what I wanted it to look at. Just - (ignore) out whatever memory
loading function you have that is loading the 2GB of memory (assuming
that's not where you're using memory you don't know about).

"top [--cum] [-ignore1] [-ignore2]"

[Rémy Oudompheng]

Le 29 oct. 2013 01:11, "Aaron Blohowiak" <aaron.b...@gmail.com> a écrit :
>
> You should do the GOGC=off -memprofilerate=1 trick for a *single* run of your benchmark and then use the resulting memory profile which will show all allocations performed ever (because the heap is never GC'd)
>

The memory profile already counts all allocations, there is no need to disable GC.

Rémy.

[Bruno Albuquerque]

In the end, what helped me was setting runtime.MemProfileRate to 0 at startup and set it to its default value before the section I wanted to track (found that out thanks to Péter Szilágyi). This seems to give me what I want.

Thanks for all the help.