heap pprof profile showing only 10Gi out of 50Gi of actually used memory
128 views
Skip to first unread message
Frank Flipper
Feb 20, 2024, 8:22:50 AMFeb 20
to golang-nuts
I have an app that's put inside k8s container and is being monitored in grafana. Memory usage graph shows me that one of the pods is using 50Gi of memory, but when I go to <pod_ip>/debug/pprof/heap I only see that it's using only 10-11Gi. I'm aware that pprof is not meant to track and show every single allocation, but I'm bothered by difference in 40Gi.
Kurtis Rader
Feb 20, 2024, 1:38:35 PMFeb 20
to Frank Flipper, golang-nuts
Is your app built with CGO? That is, do you link it against any C/C++ code that might be calling malloc?
On Tue, Feb 20, 2024 at 5:22 AM Frank Flipper <frankf...@gmail.com> wrote:
I have an app that's put inside k8s container and is being monitored in grafana. Memory usage graph shows me that one of the pods is using 50Gi of memory, but when I go to <pod_ip>/debug/pprof/heap I only see that it's using only 10-11Gi. I'm aware that pprof is not meant to track and show every single allocation, but I'm bothered by difference in 40Gi.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8eb463f6-e608-40b0-8df5-c497e2a1414an%40googlegroups.com.
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank
Frank Flipper
Feb 21, 2024, 1:43:00 PMFeb 21
to golang-nuts
it's a purely golang app
Amnon
Feb 22, 2024, 1:21:56 AMFeb 22
to golang-nuts
Worth studying https://tip.golang.org/doc/gc-guide
The heap profile will show you the heap memory in use,
but the OS process info will tell you the total process size, including memory in use,
and memory GC which has been marked as free and reclaimed by the GC, but not relinquished to the OS.
When the GC frees memory, it often does not return it to the OS, but keeps it around because it might be needed soon.
You can only return entire pages to the OS, so the GC often can't return a memory page even if most of it is free,
but a small part of it is still in use.
The heap profile will show you the heap memory in use,
but the OS process info will tell you the total process size, including memory in use,
and memory GC which has been marked as free and reclaimed by the GC, but not relinquished to the OS.
When the GC frees memory, it often does not return it to the OS, but keeps it around because it might be needed soon.
You can only return entire pages to the OS, so the GC often can't return a memory page even if most of it is free,
but a small part of it is still in use.
Reply all
Reply to author
Forward
0 new messages