Understanding Growing Memory Usage

[Matt Silverlock]

Hi all,

I'm trying to understand how App Engine Flexible reports memory usage (under the graph/metric of the same name) - specifically in an effort to figure out what leads it to growing over time.

• It's a Go application in development that serves a simple JSON API & runs background work via a multi-channel pipeline at $INTERVAL
  
• Over the last 10 hours (fresh deploy), usage has grown from 362MB to 487MB
• Over the five days prior, it grew from 440MB to 611MB
• Over the four days prior to that, it grew from 422MB to 554MB

My concern is that I'll hit the resource limit, despite the application consuming a consistent amount of memory itself. I've confirmed this by exposing Go's pprof profiles, so I can track goroutines & runtime.Memstats over time vs. the App Engine console reports.

• In the most recent case (362MB -> 487MB in 10 hours), there are no goroutine leaks nor does the total RSS or heap allocation from the OS grow - in fact, it consumes a tiny amount of memory (as expected: it's designed to!)
• Total RSS (Sys) doesn't grow - Go's runtime has allocated what it needs and hasn't needed to grow that set
• HeapAlloc isn't growing noticeably - and certainly not in-line with the 34% increase that the Memory Usage chart shows

Question: What is the Memory Usage reporting? Clearly, it's the entire system (OS + container daemon + my application), but why it is growing? Bug in the reporting? Bug in the Flex env?

---

The latest runtime.Memstats:

# runtime.MemStats
# Alloc = 3656512
# TotalAlloc = 391922816
# Sys = 8722680
# Lookups = 4635
# Mallocs = 6187675
# Frees = 6145212
# HeapAlloc = 3656512 // 3.65MiB
# HeapSys = 5767168
# HeapIdle = 606208
# HeapInuse = 5160960
# HeapReleased = 393216
# HeapObjects = 42463

~5 hours ago:

# runtime.MemStats
# Alloc = 2643608
# TotalAlloc = 213251808
# Sys = 8722680
# Lookups = 2519
# Mallocs = 3391493
# Frees = 3364582
# HeapAlloc = 2643608 // 2.64MiB
# HeapSys = 5799936
# HeapIdle = 1089536
# HeapInuse = 4710400
# HeapReleased = 155648
# HeapObjects = 26911

~10 hours ago:

# runtime.MemStats
# Alloc = 3093512
# TotalAlloc = 181426528
# Sys = 8722680
# Lookups = 2206
# Mallocs = 2832891
# Frees = 2799241

# HeapAlloc = 3407448 // 3.40MiB
# HeapSys = 5767168
# HeapIdle = 786432
# HeapInuse = 4980736
# HeapReleased = 0
# HeapObjects = 37674

[Yannick (Cloud Platform Support)]

If it isn't caused by your application, such a growth in memory consumption sounds like a bug that needs to be properly investigated. In order to do so, could you please create a private issue on the Issue Tracker and provide your project number, affected versions and any other relevant information that could allow support to reproduce the issue?

[Matt Silverlock]

Will do, thanks Yannick.

(I'd posted to see if any others had run into this, so if anyone has seen these issues too, please speak up!)