How to view heap dump?

[kant kodali]

Hi All,

I am trying to debug a memory leak and I tried several things with pprof however given that its a sampler profiler I am unable to see where the problem is, when I run linux top I can clearly see my application is maxing out of memory so I tried extracting heapdump using debug.writeHeapDump and I see a 20GB file which go tool pprof cannot open amd I unable to find any tool that can show all the allocations or the contents of the allocations. Any suggestions?

Thanks

[Robert Engels]

Take the heap dump much sooner when it is smaller. Look at the object histogram. Look for retained objects you do not expect. 

On Mar 16, 2022, at 7:16 AM, kant kodali <kant...@gmail.com> wrote:



Hi All,

I am trying to debug a memory leak and I tried several things with pprof however given that its a sampler profiler I am unable to see where the problem is, when I run linux top I can clearly see my application is maxing out of memory so I tried extracting heapdump using debug.writeHeapDump and I see a 20GB file which go tool pprof cannot open amd I unable to find any tool that can show all the allocations or the contents of the allocations. Any suggestions?

Thanks

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CA%2BiiNx8RVpN%3D3Q%3D-VLuh4pX8H-N5Pedmb6JLau0woMP9zq0WCg%40mail.gmail.com.

[kant kodali]

That’s going to be tricky but I will try since pprof can’t seem to open over 500MB on my laptop. Is the command still go tool pprof heapdump.out? Where heapdump.out is the output file generated from debug.write heapdump 

Sent from my iPhone

On Mar 16, 2022, at 5:41 AM, Robert Engels <ren...@ix.netcom.com> wrote:



[Josh Bleecher Snyder]

IIRC the heap dump viewer is maintained by Keith out of tree. Maybe it's https://github.com/randall77/hprof? See also https://go.googlesource.com/proposal/+/master/design/16410-heap-viewer.md.

It might be easier for you to set runtime.MemProfileRate to 1, so that every allocation gets recorded. Then run for a while, call runtime.GC to force a GC, and pass -inuse_space or -inuse_objects to pprof to see what objects are still in use.

Hope that helps.

-josh

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/605B49CB-D763-45D9-BA39-8195EB03300C%40gmail.com.

[t hepudds]

Hi all,

I'm not sure that hprof still works. There was some work started in #16410 to update it or create a new heap dump viewer, but to my knowledge that did not complete.

golang.org/x/debug/cmd/viewcore is another option.  I think the version at master also does not work with recent Go versions, but the nice folks at Cockroach Labs have sent some patches to get it to work with more recent releases.  In theory, it should work with Go 1.16 and 1.17 with those patches, though I have had more success with Go 1.16.

FWIW, I think viewcore is an underutilized tool, and it would be nice to get it into shape, though of course there are many balls in the air.

Here is a from scratch example of installing & using viewcore to check:

    * where memory is being used in a test program, and

    * an example of checking why a particular object is reachable.

    # install go1.16
    go install golang.org/dl/go1.16.15@latest
    go1.16.15 download
   
    # build viewcore with cockroachdb pull request
    git clone https://github.com/golang/debug
    cd debug
    git fetch origin pull/7/head:pull-7
    git checkout pull-7
    go1.16.15 install ./cmd/viewcore
    cd ..
   
    # build & run a test program that allocates
    # a configurable amount of heap
    export GOTRACEBACK=crash
    go1.16.15 install -gcflags='all=-N -l' github.com/thepudds/arena-performance/cmd/binarytree-original@latest
    binarytree-original 24 &   # will run for ~20 sec, using ~2.3 GB RSS
   
    # create a core
    kill -SEGV $!              # pid of binarytree-original background process
    ls -l ./core
   
    # view it with viewcore
    viewcore ./core
   
    # explore the core
    (viewcore) histogram --top 5
        count  size     bytes type
     60279514    16 964472224 unk16
            8 10240     81920 runtime.p
           45   384     17280 runtime.g
            9  1024      9216 runtime.m
            1  8192      8192 [34+5?]bucket<string,*unicode.RangeTable>

    (viewcore) reachable c00031d580
    main.Run.func1
    main.NewTree
    main.NewTree.unk →
    c06dea4710 unk16 f0 → f0
    c0425c8d40 unk16 f0 → f0
    c01426eb70 unk16 f0 → f0
    c00a0aac60 unk16 f0 → f0
    c0058748a0 unk16 f8 → f0

That was on Debian 11 (bullseye).

Regards,

--thepudds

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CAFAcib_ego%2BQBnf7PWDzC0XrFidNASBraAD8kfnWsmERoFt%2B3w%40mail.gmail.com.