Slow memory increase with file access log to an actual file instead of stdout

[premkumar bhaskal]

Hi Team,

I am using istio 1.20 having envoy 1.28 and recently we switched the file access log from /dev/stdout to an actual file (a volume mount).

What I am seeing is that slowly memory has been increasing, and it is proportional to the requests rate as well as the content logged in access log. 

We confirmed by going back to /dev/stdout (and even /dev/null) that writing to an actual file is an issue and causes slow increase in memory.

I think I am hitting this particular issue https://github.com/envoyproxy/envoy/issues/9458 (access log: file access log memory growth between flush events)

Is there a plan to address this? Is there a plan to make  `MIN_FLUSH_SIZE` configurable?


Thanks,
Prem

[premkumar bhaskal]

Also i see that even though the metric heapster.pod_container.memory.working_set shows increase in usage, but heapster.pod_container.memory.rss is flat and not increasing.

[Yan Avlasov]

Nobody is working on this issue. If you have ability or resources, we will welcome this change as a contribution.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/envoy-users/a8b2b03d-fd4d-47e5-8544-0e486b8d8b3dn%40googlegroups.com.

[premkumar bhaskal]

Thank you for reply.

Could you please confirm my analysis, if we write to a file (may be on slower filesystem), the memory increase is expected?

Thanks,

Prem

[Yan Avlasov]

Yes, it is possible that memory consumption will grow with slow file writes.

To view this discussion visit https://groups.google.com/d/msgid/envoy-users/aa8c21af-849a-49b7-8ac4-542fd621fb7bn%40googlegroups.com.