Memory usage and go tool pprof

[Alexander Petrovsky]

Hello!

I'm a golang newbie, so could anyone help me with strange memory allocation by my program.

What I have from "runtime/pprof":

Entering interactive mode (type "help" for commands)
(pprof) top
25987.89MB of 26181.92MB total (99.26%)
Dropped 59 nodes (cum <= 130.91MB)
      flat  flat%   sum%        cum   cum%
23691.69MB 90.49% 90.49% 24513.87MB 93.63%  calc/reference.FetchCost
 1536.02MB  5.87% 96.36%  1562.52MB  5.97%  calc/reference.FetchRevenue
  760.17MB  2.90% 99.26%   760.17MB  2.90%  calc/reference.kvsCost
         0     0% 99.26% 26181.92MB   100%  runtime.goexit
         0     0% 99.26% 24513.87MB 93.63%  calc/reference.Update.func1
         0     0% 99.26%  1562.52MB  5.97%  calc/reference.Update.func2

But, atop/htop/top/ps shows me the follow:

# ps aux | head -n 1; ps aux | grep calc
USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      69439  129 96.0 64702588 63118748 ?   Ssl  00:10 806:18 /opt/calc/bin/calc --config /opt/calc/conf/prod.yml

And sometimes comes in OOM killer and kill my program:

Nov 20 06:29:34 mcalc-stage kernel: Out of memory: Kill process 61018 (calc) score 941 or sacrifice child
Nov 20 06:29:34 mcalc-stage kernel: Killed process 61018 (calc) total-vm:65761252kB, anon-rss:64416000kB, file-rss:128kB
Nov 20 06:29:34 mcalc-stage kernel: mcalc: page allocation failure: order:0, mode:0x2015a 

So, the main question why pprof show me that my program eats only 25Gb memory, while the atop/htop/top/ps shows me real situation.

--

Петровский Александр / Alexander Petrovsky,

Skype: askjuise

Phone: +7 914 8 820 815

[Tamás Gulácsi]

What is "real"?
65Gb has been asked from the OS, and it gave that much to the runtime. But it used only 25Gb of it.
Maybe even released some to the OS, but maybe that did not took that.

Is the 25Gb really needed and used for good? The OOM suggests that for moments your program allocated much more....

[Alexander Petrovsky]

Hello!

"Real" is what I see in atop/htop/ps, as i mentioned erlier.

Yep, about asked memory from OS is in my mind too, but how can I find situation when/why it's occur?

To be more precise, my app load N merabytes from DB, and put them into map. So, I fully understand when N megabytes transform to N gigabytes in memory in map, in the next iteration, after some time, my app doing load and transform again, so, I can imagine situation, when in exist two maps, N gigabytes each, but after that old map is gone, and my app start using new map! So, in my mind GC shoud kill old map, is I'm right? I How can I tune memory allocator? May be I can tune it to allocate memory more tight? Why my app don't return back allocated memory?

вторник, 22 ноября 2016 г., 0:10:20 UTC+3 пользователь Tamás Gulácsi написал:

[Dave Cheney]

Please turn on gc debugging with end GODEBUG=gctrace=1 and check that your applications heap usage is behaving as you believe.