Unable to use net/http/pprof to profile live webserver

[569...@gmail.com]

Hello,

I have a webserver application written in Golang that runs on a dedicated server. It serves a popular website and uses almost 100% CPU and 100GB of memory. It's in production but I'd like to profile it as it is: live. It's on a remote webserver, which I have access too via HTTP and SSH (i.e. non-local.)

I'd like to profile the webserver so that I can optimize both CPU and memory usage. I tried adding "import _ net/http/pprof" but I am unable to access any of the reports.

The website is served using the "github.com/facebookgo/grace/gracehttp" package to allow for graceful restarts, as follows:

 serverHTTPS := &http.Server{
 Addr:           constants.URL + `:443`,
 Handler:        hdl,
 ReadTimeout:    30 * time.Second, // how long a file can be uploaded for before it timesout
 WriteTimeout:   60 * time.Minute, // how long a file can be downloaded for before it timesout
 MaxHeaderBytes: 1024 * 1024, // 1 MB
 TLSConfig: &tls.Config{Certificates: []tls.Certificate{cert}},
 TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler)), // disable http2 due to bugs
 }
 serverHTTP := &http.Server{
 Addr:           constants.URL + `:80`,
 Handler:        redirhdl{},
 ReadTimeout:    10 * time.Second, // how long a file can be uploaded for before it timesout
 WriteTimeout:   10 * time.Minute, // how long a file can be downloaded for before it timesout
 MaxHeaderBytes: 512 * 1024, // 512 KB
 }
 log.Fatal(gracehttp.Serve(serverHTTPS, serverHTTP))

The "hdl" struct has a ServeHTTP method which parses the URL to decide which page to serve.

If I try to visit the site with /debug/pprof all I get is my standard 404 page not found message.

I tried catching the /debug/pprof path from ServeHTTP and just returning this immediately, but that doesn't work either, it just gives a blank page.

Any ideas how I can profile this live webserver?

Thanks,

Alasdair

[Shawn Milochik]

Have you tried just running a separate `ListenAndServe` on another port on localhost, such as "localhost:6060"?

That is what's done in the documentation and I've copied that with success many times.

    // should do the trick

    go http.ListenAndServe("localhost:6060", nil)

Then (from within the server) you should be able to hit:

http://localhost:6060/debug/pprof/heap

http://localhost:6060/debug/pprof/profile

http://localhost:6060/debug/pprof/goroutine?debug=2

http://localhost:6060/debug/pprof/goroutine?debug=1

If you wget or curl the first two, you can download them to your local machine and use them with the go tool to view the nice images and see things like allocations by lines of code:

go tool pprof binaryname filename.pprof

Commands I've found most useful:

web

top 10

list <function name>

[ma...@influxdb.com]

Besides the HTTP standard approach and the workaround that Shawn mentioned, a couple other options include:

* installing the gops agent [1] in the binary and using the gops CLI when SSHed onto the machine

* adding your own hook (via CLI, custom HTTP endpoint, on a timer, etc.) to directly capture profiles using runtime/pprof [2] or the simpler lightweight wrapper in pkg/profile [3] 

You're right that profiling the live app will give you the most informative results, but if you're new to profiling, it's worth running locally first to gain familiarity and comfort with the tooling, where you can quickly make changes to your application to see how the profile results change.

[1] https://github.com/google/gops

[2] https://golang.org/pkg/runtime/pprof/

[3] https://github.com/pkg/profile