Why HeapInuse - HeapReleased is greater than "used" memory returned by "free" util?

[Sergey Naumov]

Hello.

I've originally asked this question on StackOverflow: https://stackoverflow.com/questions/57714300/ho-to-properly-interpret-heapinuse-heapidle-heapreleased-memory-stats-in-gol

So here I will write just a quick summary:

I monitor memory usage of my program to determine memory pressure conditions when I need to clean up some internal cache to prevent my program from being killed by OOM killer.

The problem is that (HeapInuse - HeapReleased) value is generally in line with other system memory stats, but sometimes (after Go runtime starting to release some used memory to OS), this resulting value is greater than "used" memory reported by "free" utility:

# Looks BAD: 13488 + 14 + 3631 - 489 = 16644 - more that total system memory
Available: 13488M, HeapAlloc: 10M, HeapInuse: 14M, HeapIdle: 3631M, HeapReleased: 489M

# It is strange that at this moment HeapIdle - HeapReleased = 3142
# > than 2134M of used memory reported by "free" utility.
$ free
              total        used        free      shared  buff/cache   available
Mem:       16123232     2185696    13337632         200      599904    13621988
Swap:      73242180       34560    73207620

But, after a while, everything is OK again:

# Looks good: 11815 + 2325 + 1320 = 15460
Available: 11815M, HeapAlloc: 2322M, HeapInuse: 2325M, HeapIdle: 1320M, HeapReleased: 0M

I do not understand from where this additional 1.2G (16644M - 15460M) of memory comes from. When total sum is around 15,5G everything looks good and memory stats from Go and from "free" are properly aligned. But when total sum is aroung 16.6G - they aren't - as total memory size is less than that. From where this 1.2G of memory originates?

Used swap size remained the same during the whole test.

Thanks in advance,

Sergey.