Help me understand pprof to detect memory leak

[Albert Tedja]

I am trying to see if there's a leak in my Go programs, but I can't quite tell if it is.  It's an app that is serving TCP traffic reading JSON payloads.

When I enabled GODEBUG=gctrace=1, I am seeing this:
gc 19 @689.729s 0%: 0.049+0.41+0.020+0.40+0.33 ms clock, 0.39+0.41+0+0/0.64/1.2+2.6 ms cpu, 2->2->0 MB, 4 MB goal, 8 P
GC forced
gc 20 @809.733s 0%: 0.045+0.43+0.001+0.50+0.25 ms clock, 0.36+0.43+0+0/0.62/1.0+2.0 ms cpu, 0->0->0 MB, 4 MB goal, 8 P

So it says 2->2->0 MB.  And following that is 0->0->0MB.  So GC does its job, right?

But top VIRT, RES, and SHR are not dropping at all.
 9351 me    20   0  750744  16656   4084 S   0.0  0.2   0:01.52 myapp

pprof is somehow showing that heap isn't declining

profiles:

0	block
95	goroutine
60	heap
26	threadcreate

After I hammer the program with lots of traffic, the number of goroutines would go back to normal once those connections are dropped, but heap and threadcreate stays, and only increases as new traffic comes in.

If I click on heap, it's basically populated by encoding/json stuff.  Is the encoding/json library leaking?

0: 0 [1: 16] @ 0x40e1b9 0x569767 0x568f93 0x56950c 0x568fbb 0x56950c 0x564138 0x56255d 0x564ea9 0x56255d 0x561a06 0x575644 0x47c115 0x47b5bb 0x4607e1
#	0x569767	encoding/json.(*decodeState).literalInterface+0x1a7	/usr/local/go/src/encoding/json/decode.go:946
#	0x568f93	encoding/json.(*decodeState).valueInterface+0x53	/usr/local/go/src/encoding/json/decode.go:845
#	0x56950c	encoding/json.(*decodeState).objectInterface+0x21c	/usr/local/go/src/encoding/json/decode.go:909
#	0x568fbb	encoding/json.(*decodeState).valueInterface+0x7b	/usr/local/go/src/encoding/json/decode.go:843
#	0x56950c	encoding/json.(*decodeState).objectInterface+0x21c	/usr/local/go/src/encoding/json/decode.go:909
#	0x564138	encoding/json.(*decodeState).object+0x408		/usr/local/go/src/encoding/json/decode.go:505
#	0x56255d	encoding/json.(*decodeState).value+0x3bd		/usr/local/go/src/encoding/json/decode.go:296
#	0x564ea9	encoding/json.(*decodeState).object+0x1179		/usr/local/go/src/encoding/json/decode.go:613
#	0x56255d	encoding/json.(*decodeState).value+0x3bd		/usr/local/go/src/encoding/json/decode.go:296
#	0x561a06	encoding/json.(*decodeState).unmarshal+0x196		/usr/local/go/src/encoding/json/decode.go:157
#	0x575644	encoding/json.(*Decoder).Decode+0x274			/usr/local/go/src/encoding/json/stream.go:67

Zabbix monitoring is showing that free memory is declining very very slowly over a long period of time.  The memory usage didn't even seem to recover even when I restarted my program.

I am starting to believe some else is causing the leak, but I can't be quite sure now.  Any pointers?

[Dave Cheney]

Based on the data you have shown you don't have a memory leak. Look for a line marked scvg: that reports the absolute amount of memory requested from the operating system. Trust it. If the operating system is reporting larger numbers its accounting is wrong.