Memory pressure in an embedded linux environment

[ive...@noisolation.com]

Hello!

I'm having problems using the Chromium in an embedded device, due to what seems to be too aggressive memory consumption.

Architecture: aarch64 (armv8-a).

OS: Linux (kernel 4.9.39).

Test situation:

Chromium Embedded Framework (CEF; cefsimple example program) running with a cgroup limiting memory to a max of 320MB. If I hadn't had this cgroup limit the same happens only much later when the whole ram is spent (1GB). A simple html+images+js webpage that uses js to switch between images in a loop. No new content over time, just the same over and over again.

Expected:

Chromium to consume the size of all binaries as the memory mapped libraries are used and loaded into ram + os process/thread overhead + webpage content size ++.

What actually happens:

Chromium uses all available memory to within 1MB, not right away but it slowly grows from half of the given max until there is nothing left, then hugs that limit so closely that either it gets itself killed by the OOM, or if the OOM is turned off it fails to create new threads (pthread_create) due to those resource limits. This depends a bit on the type of content. With the image switcher described above it grows in steps, which if graphed look like stairs. However, if the page has a row of images and use CSS animations to translate them back and forth the memory "leak" / usage grows slow and steady, completely linearly, until there is nothing left and it hugs that limit closely until it fails.

Questions:

1. Is there a concept of Memory Pressure for Linux? I cannot see it in the source. I assume this is supposed to hint to Chromiums allocators to ease off and GC when closing on the limit.
2. Why does Chromium insist on using all the memory? It is clearly more than it needs since it spends some time getting to the max.
3. How can I make Chromium believe that it has 20-30MB less memory than it does by patching the source? I'm thinking this may be enough to hide the crash since it does seem to respect the limit somewhat, just not well enough.
1. I'm hoping that when it over-steps the allocators will GC to correct itself. Might be wrong?
2. Keeping it a bit below the limit will also prevent the kernel from slowing the CPU to punish the process for using all its allowed memory (high vs. max memory limits in cgroups).
4. Any other ideas for me?

I really appreciate any help or guidance you may provide me with!

[Mario Sanchez Prada]

Hi

I don't have all the answers to your questions and it's even possible that what I'm going to comment is a bit outdated or plainly wrong, so please anyone else with more knowledge on this do correct me if needed.

In any case, here it goes anyway in case it can be of any help for your particular use case:

On 26/11/18 16:49, ive...@noisolation.com wrote:
> [...]
> *Questions:*
> 1. Is there a concept of Memory Pressure for Linux? I cannot see it in the source. I assume this is supposed to hint to Chromiums allocators to ease off and GC when closing on the limit.

As far as I know there's still the old system based on the MemoryPressureMonitor class in place under //base/memory [1], which is still in use in some platforms but unfortunately not in Linux: just on Windows, Mac and ChromeOs.

Also as far as I remember, there have been attempts in the past to refactor this code into a |memory_pressure| component under //components/memory_pressure, including a Linux implementation (see [2]). However, and this is why you can't find anything in the sources, checking the commit history [3] I found out that this has been abandoned like 1.5 years ago in favour of the "memory coordinator work" (see [4]).

So, I think I'd suggest getting familiar with that "memory coordinator" work and figuring out the status of it for Linux, and whether you could work in that area to get your use case supported. Mind you, I'm not familiar with that so please take my word with a grain of salt, you'd better get in touch with the right people there and get advice from them if this is something you consider exploring.

> [...]
> 4. Any other ideas for me?

Alternative, you could follow another approach and do what Endless have done downstream to support their low-memory ARM devices a long time ago: hack your own version of the old MemoryPressureMonitor for your platform and adapt it to your needs. Endless did it based on ChromiumOS' implementation of MemoryPressureMonitor here, see the main commits here:

https://github.com/endlessm/chromium-browser/commit/6196cc79
https://github.com/endlessm/chromium-browser/commit/b6df3633
https://github.com/endlessm/chromium-browser/commit/146408fd

(you might find other interesting commits in the whole set of downstream patches in [5])

However, this came not without problems (e.g. heuristics not being entirely right, memory still filing up in some configurations...) so, based on my own experience and if it's feasible for you, I think I'd advise you to first check the status "memory coordinator" work first and see if that can fit your needs, before going the "downstream-only" route :-).

> I really appreciate any help or guidance you may provide me with!

As I said a couple of times already, take this mail with a grain of salt, but other than that I hope this is useful in some way.

Thanks,
Mario

[1] https://cs.chromium.org/chromium/src/base/memory/memory_pressure_monitor.h
[2] https://bugs.chromium.org/p/chromium/issues/detail?id=463606
[3] https://chromium.googlesource.com/chromium/src/+log/master/components/memory_pressure/direct_memory_pressure_calculator_linux.cc
[4] https://chromium.googlesource.com/chromium/src/+/4d1357d65728d99aef9de10f4ab8a914cb09b96d
[5] https://github.com/endlessm/chromium-browser/commits/eos3.5

[ive...@noisolation.com]

Thanks Mario.

I will look into those references and patches tomorrow. I discovered earlier today from the source that linux fetches its memory stats from "/proc/meminfo", and thus probably knows nothing about the cgroup limits I've given it. Which could explain why it seemingly goes CPU to a 100% when it reaches the limit and actively hugs the line (I'm assuming a loop trying to malloc somewhere, or similar).

My plan right now is to reimplement it to get its memory stats info from the cgroup stats instead, and check how that effects Chromium :).

[Luke Halliwell]

Btw, we face a similar issue on Chromecast.  Currently we run our own memory pressure monitor and generate pressure signals, without which we would hit OOM crashes pretty quickly.  You can find this code under chromecast/ dir.

Some possible downsides to look out for:

1) memory pressure signals can cause some extra work and possible jank (e.g. GC pauses, and sometimes dropping compositor tiles which need to be regenerated immediately)

2) code can get added anywhere to respond to memory pressure signals, so it can be an ongoing maintenance issue to look out for unexpected behaviour changes (e.g. I remember one where idle sockets got closed that caused us problems, you'll see we have some code hanging around still to deal with that).