What is pprof overhead like when gathering a profile?

1034 views
Skip to first unread message

nat...@honeycomb.io

unread,
Jul 24, 2017, 8:44:10 PM7/24/17
to golang-nuts
Hello,

I am curious what the performance impact of running pprof to collect information about CPU or memory usage is. Is it like strace where there could be a massive slowdown (up to 100x) or is it lower overhead, i.e., safe to use in production? The article here - http://artem.krylysov.com/blog/2017/03/13/profiling-and-optimizing-go-web-applications/ - suggests that "one of the biggest pprof advantages is that it has low overhead and can be used in a production environment on a live traffic without any noticeable performance penalties". Is that accurate?

Thanks!

Nathan

Dave Cheney

unread,
Jul 24, 2017, 8:53:57 PM7/24/17
to golang-nuts
Here's an entirely unscientific method to determine the overhead of profiling. The Go distribution contains a set of basic benchmarks, one of which is a loopback based http client server benchmark. Running the benchmark with and without profiling gives a rough ballpark for the overhead of profiling.

lucky(~/go/test/bench/go1) % go test -run=XXX -bench=HTTPClientServer
goos: linux
goarch: amd64
BenchmarkHTTPClientServer-4        20000             84296 ns/op
PASS
ok      _/home/dfc/go/test/bench/go1    4.274s
lucky(~/go/test/bench/go1) % go test -run=XXX -bench=HTTPClientServer -cpuprofile=/tmp/c.p
goos: linux
goarch: amd64
BenchmarkHTTPClientServer-4        20000             85316 ns/op
PASS
ok      _/home/dfc/go/test/bench/go1    4.402s

You could use this to experiment with the other kinds of profiles; memory, block, trace, etc.

If you wanted to go a step further you could adding profiling to your own project with my github.com/pkg/profile package then compare the results of a http load test with and without profiling enabled.

Thanks

Dave

j...@google.com

unread,
Jul 24, 2017, 8:55:42 PM7/24/17
to golang-nuts
It would be very speculative to provide reference numbers without actually seeing the specific program. You can benchmark the latency/throughput with the CPU profiler on to see a realistic estimate. FWIW, memory profiling, goroutine, thread create profiles are always on. At Google, we continuously profile Go production services and it is safe to do so.

Dave Cheney

unread,
Jul 24, 2017, 9:11:36 PM7/24/17
to golang-nuts
Another option is to profile a % of requests. In the past I've done that by enabling profiling on a set % of application servers then extrapolating from there.

Jaana Burcu Dogan

unread,
Jul 25, 2017, 1:16:58 AM7/25/17
to Dave Cheney, golang-nuts
What we do is to periodically turn on profiling for X seconds, collect some data and turn it off again. We do it at every once a while periodically. We target a single or a group of instances in a large group.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/e6lB8ENbIw8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages