Help with runtime/pprof

[beatgammit]

I'm running a server on an embedded ARM device that runs out of memory after about 10 hours or so. The server produces data in response to http requests, and only stores a portion of that data in memory. I've watched it with htop and it seems like the memory usage is steadily increasing. I'd like to see where this is happening over time.

I have something like this for debugging right now:

import "runtime/pprof"

func main() {
    go func () {
        for {
            pprof.WriteHeapProfile(os.Stdout)
            time.Sleep(time.Second * 10)
        }
    }()

    // start server stuff
}

However, this give me this rather unhelpful output:

heap profile: 0: 0 [0: 0] @ heap/1048576

I'd like to get something that prints out the places that use the most memory, because I have a suspicion that it is a growing map problem. I'll also be checking out the number of Goroutines and print out a goroutine stack trace.

Since I'll be monitoring this over a long period of time, I figured printing to standard out (which is redirected to a file) would be the safest way to do it. I already have a mechanism that restarts the server and saves old logs on failure, so when there is an out of memory exception, I can see the most recent logs.

I also tried doing something like this:

    go func () {
        http.ListenAndServe(":6060", nil)
    }()

And that did allow me to go to localhost:6060/debug/pprof/heap and get data manually (with slightly more helpful output), but it seems to only show allocs, not current heap objects. I'm pretty new to profiling under Go as you can probably tell.

I'm pretty much looking for something similar to Chrome's JavaScript heap snapshot in the developer tools, which shows the size of each object at that specific point in time.  How can I get this data? I'm ok with making a cron job or something to curl a resource once an hour or so.

Thanks!

Jameson

[beatgammit]

I got some more information by using Lookup("heap").WriteTo(os.Stdout, 1), but this still doesn't seem to show all the information that I'm looking for. Maybe I just don't know how to interpret it.

I'm looking for a complete dump of what's in the heap and the size of each object (like v8's heap dump in Chrome's developer tools with heap snapshot).

[Rémy Oudompheng]

On 2012/8/22 beatgammit <beatg...@gmail.com> wrote:
> I got some more information by using Lookup("heap").WriteTo(os.Stdout, 1),
> but this still doesn't seem to show all the information that I'm looking
> for. Maybe I just don't know how to interpret it.
>
> I'm looking for a complete dump of what's in the heap and the size of each
> object (like v8's heap dump in Chrome's developer tools with heap snapshot).

The preferred way of using the heap dumps is through the pprof utility.
It gives you human-readable stats and graphs.

When using Lookup("heap").WriteTo(os.Stdout, 1) you get additional
readable information in the dump, but pprof is better at giving that.

See: http://blog.golang.org/2011/06/profiling-go-programs.html

Rémy.

[beatgammit]

Thanks for the reply!

I've tried using the pprof utility, but I'm not sure how it works. I tried this:

// instead of the manual pprof.Lookup... stuff

go func () {

    http.ListenAndServe(":6060", nil)

}()

go tool pprof path/to/server/binary http://localhost:6060/debug/pprof

go tool pprof: exit status 1

However, I get this:

http://localhost:6060/debug/pprof: No such file or directory

I have a server running on localhost:5000 and this on 6060

If I start the server first, I get an error saying:

FATAL ERROR: Did not specify profile file

I couldn't find any documentation on what a profile is or looks like, although golang.org/pkg/runtime/pprof hints at one.

If I navigate to localhost:6060/debug/pprof, I get the expected page, however I don't know how to get complete heap dumps (I just get what Lookup("heap").WriteTo(os.Stdout, 1) gives).

[Rémy Oudompheng]

On 2012/8/22 beatgammit <beatg...@gmail.com> wrote:
> Thanks for the reply!
>
> I've tried using the pprof utility, but I'm not sure how it works. I tried
> this:
>
> // instead of the manual pprof.Lookup... stuff
> go func () {
> http.ListenAndServe(":6060", nil)
> }()
>
> go tool pprof path/to/server/binary http://localhost:6060/debug/pprof
> go tool pprof: exit status 1
>
> However, I get this:
>
> http://localhost:6060/debug/pprof: No such file or directory
>
> I have a server running on localhost:5000 and this on 6060
>
> If I start the server first, I get an error saying:
>
> FATAL ERROR: Did not specify profile file
>
> I couldn't find any documentation on what a profile is or looks like,
> although golang.org/pkg/runtime/pprof hints at one.
>
> If I navigate to localhost:6060/debug/pprof, I get the expected page,
> however I don't know how to get complete heap dumps (I just get what
> Lookup("heap").WriteTo(os.Stdout, 1) gives).

You need to do:

go tool pprof http://localhost:xxx/debug/pprof/heap

Heap profiling only gives you counts of allocated items fer call site
(currently used + total ever allocated). By default, it doesn't count
any items, but just an appropriate fraction of them, and the pprof
utility recomputes an estimate of how many items there are actually.

That kind of profiling is usually enough for common debugging tasks.


Rémy.

[beatgammit]

Oh my goodness!! Many thanks!!

I didn't have the /heap at the end, so I guess it didn't know what to do. Now I have the interactive profiler, and I am very happy. Thank you so much!

Jameson