Seeing high RES memory but very little inuse_space

[Mohit Goyal]

We have a job executor that runs periodic jobs. We have being seeing a constant growth in the RES memory (to the tune of 10 GB). However, our heap usage is constant at around 75-85MB. Here are the stats for the same - 

• process_resident_memory_bytes 1.0107805696e+10 = 10GB
• go_memstats_heap_idle_bytes 1.00286464e+08     = 100MB
• go_memstats_heap_inuse_bytes 1.01433344e+08    = 100MB
• go_memstats_alloc_bytes 9.0241416e+07          = 90MB 

• go_gc_cycles_automatic_gc_cycles_total         = 8249
• go_gc_heap_allocs_bytes_total 9.9512908776e+10 = 99 GB
• go_gc_heap_frees_bytes_total  9.942266736e+10  = 99GB

The go version is 1.19.1. Looking at the call graph, we see that json to struct conversion has been allocating a lot of space (40GB out of 99GB). However, its only seen in the alloc_space and not in the inuse_space. Any pointers to what could be the reason and if its anything you have also seen would be helpful. 

[Mohit Goyal]

Note: the json library is https://github.com/goccy/go-json