wild ideas around lowmemorykiller

[lhil...@mozilla.com]

A pair of wild ideas from a barely initiated n00b. I apologize if I don't have proper context or these ideas are naive. Whatever the case, I'm sure it will be educational for me.

Idea 1: as I crawl around on my ZTE open I see some suspicious configuration:

first, in /sys/module/lowmemorykiller/parameters/minfree

1024,1280,1536,1792,2048,5120

next, in /sys/module/lowmemorykiller/parameters/notify_trigger

3584

The question: Why are we notifying of memory pressure when we get down 14mb but killing off application level (adj >= 10) processes, before that, at 20mb?

The idea: Is lowmemorykilling for adj >= 10 down at 10mb undesirable for some reason? would these be better parameters which allow us to delay killing processes until we've had a chance to respond to the low memory event that raises up via GonkMemoryPressureMonitoring.cpp?

Here's what my b2g-info output looks like with updated parameters and Nuwa enabled: https://gist.github.com/lloyd/7467857

Note line 35. (also, this is awesome!)

Testing the device I find I'm able to run an additional app in that 10mb, and I feel like behavior under memory pressure is without regression.

Idea 2: I'm having a hard time tracing what lowmemorykiller notifications trigger. Is this documented anywhere?

It would seem with a notification up a little higher (perhaps at 20meg where the killing happens today) we could deliver a message to non-foreground applications that could choose to ignore it, or choose to shut themselves down cleanly and dodge the reaper.

So the idea is, should we consider application level events when we detect impending memory pressure?

thanks for listening!
lloyd

[Bob Thulfram]

I dont know the answer either but I would like to. also, how about a copy app to sd card like android - life saver with low mem phones like zte open.

ps - what os version are you running?

[Bob Thulfram]

[Gabriele Svelto]

On 14/11/2013 23:27, lhil...@mozilla.com wrote:
> A pair of wild ideas from a barely initiated n00b. I apologize if I don't have proper context or these ideas are naive. Whatever the case, I'm sure it will be educational for me.

This particular piece of functionality is quite complicated so it's not
surprising that the way some parameters were set confused you :) We (and
by we I mean I) should really make a full write-up of how the whole
chain of low-memory notification works because it's quite complex.

> Idea 1: as I crawl around on my ZTE open I see some suspicious configuration:
>
> first, in /sys/module/lowmemorykiller/parameters/minfree
>
> 1024,1280,1536,1792,2048,5120
>
> next, in /sys/module/lowmemorykiller/parameters/notify_trigger
>
> 3584
>
> The question: Why are we notifying of memory pressure when we get down 14mb but killing off application level (adj >= 10) processes, before that, at 20mb?

The reason why we do that is that whenever an application is sent into
the background we also send it a message that forces it to minimize its
memory usage, see here:

http://hg.mozilla.org/mozilla-central/file/7b014f0f3b03/dom/ipc/ProcessPriorityManager.cpp#l1024

So in general there's not much to be made by sending memory-pressure
events to background apps. They're usually already minimized which is
why we want the low-mem killer to get rid of them before we start to
send memory-pressure events.

You can find part of the discussion on why the trigger was put at that
particular level (together with some of the experiments that led to it)
in this bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=877222

> The idea: Is lowmemorykilling for adj >= 10 down at 10mb undesirable for some reason? would these be better parameters which allow us to delay killing processes until we've had a chance to respond to the low memory event that raises up via GonkMemoryPressureMonitoring.cpp?

We arrived at those parameters by testing different combinations.
They're not optimal but for now they're the best we could come up with.
One of the issues of setting the lowmemkiller parameters too low is that
they can cause instability as available memory dries up potentially
leaving enough memory only for one app, preventing everything else from
running.

> Idea 2: I'm having a hard time tracing what lowmemorykiller notifications trigger. Is this documented anywhere?

That's one of the thing I should really write somewhere :) What's
happening is that the memory pressure monitor fires some memory-pressure
events. Those are delivered to the interested objects (caches, GC,
etc...). Look for lines like this one:

http://hg.mozilla.org/mozilla-central/file/7b014f0f3b03/dom/ipc/ContentParent.cpp#l758

Those messages are also propagated from the main process to all
applications, see here:

http://hg.mozilla.org/mozilla-central/file/7b014f0f3b03/dom/ipc/ContentParent.cpp#l1967

> It would seem with a notification up a little higher (perhaps at 20meg where the killing happens today) we could deliver a message to non-foreground applications that could choose to ignore it, or choose to shut themselves down cleanly and dodge the reaper.

In general that doesn't work; when an application is starting to consume
memory it happens too fast for background applications to react in a
timely manner. Trying to minimize the memory of background applications
consumes CPU and that's also something we don't want in a number of
scenarios.

For example, when receiving an incoming call we need the dialer
application to start up fast. If there's not enough memory the best
option is to kill some other app; that saves the memory without
burdening the CPU thus allowing the user to see the incoming call ASAP.

> So the idea is, should we consider application level events when we detect impending memory pressure?

That's something I've considered but ATM there's no way to let the
content code detect those events and it's unclear if letting those
events known to the actual apps would be beneficial.

Gabriele

[Andreas Gal]

Nice analysis. Idea 1 you might be onto something. Maybe NuWa rebalanced our memory equation here? I don't know this part of the code particularly well. Paging Kyle and Nick who might know more. This code is a bit under-owned for the last months, so if you find someone suspicious, chances are, its broken.

As for your question, when the low-memory notification arrives we send a Gecko-level notification around that flushes caches (gfx caches, code caches, etc). We also flush caches when we background an application (read: page) and an event is delivered to the application to indicate that its no longer visible. The page is supposed to save all its state at that point, so I guess there isn't much need to explicitly exit. If the OOM condition gets worse the background processes will get reaped, which is not a bad thing, and suiciding them early probably doesn't benefit us much. I might be wrong. Again, I don't know this code very well, except that its super finicky and it was a pain to tune this.

Andreas

> _______________________________________________
> dev-b2g mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-b2g

[Gabriele Svelto]

On 16/11/2013 11:24, Andreas Gal wrote:
> Nice analysis. Idea 1 you might be onto something. Maybe NuWa rebalanced our memory equation here?

Yes, it would be worth to re-evaluate our parameters for NUWA as the
current settings were tailored before it. In addition to that, those
parameters were also geared for 256MiB devices that had ~180MiB of
memory available to userspace. We have a bug (886278) for re-evaluating
them for devices with more memory.

I had posted a proposal there to make those and other values dynamically
computed depending on the total amount of free memory but didn't have
time to follow-up on that:

https://bugzilla.mozilla.org/show_bug.cgi?id=886278#c10

Maybe it's time that I prioritize this stuff.

> This code is a bit under-owned for the last months, so if you find someone suspicious, chances are, its broken.

Yeah, I happen to know this part quite well and I should really be doing
a write up. I've opened a bug for it so that we can track it and I won't
forget:

https://bugzilla.mozilla.org/show_bug.cgi?id=939415

> If the OOM condition gets worse the background processes will get reaped, which is not a bad thing, and suiciding them early probably doesn't benefit us much. I might be wrong..

Actually we kill background applications before sending memory-pressure
notifications. The rationale behind it is that on the one hand
background applications have already been minimized so there's not much
to gain there and on the other hand keeping them alive might be detrimental.

The archetypal example of why it's bad to keep background apps alive is
the following: the phone's busy with lots of apps but the dialer has not
been launched yet. An incoming call arrives triggering the dialer to
start up, this will cause a spike in memory consumption as we make room
for it.

If we send a memory-pressure notification at this stage we will: a)
potentially save busy apps which might load the CPU at a time where we
need to react fast, b) generate a storm of CPU activity as all
background apps wake up and run the GC and finally c) slow down the
dialer by telling it to also run the GC*

The end result is that we'll often miss the incoming call. Killing the
background apps on the other hand decreases the chance of CPU time and
memory being diverted from what most needs it.

Dialer aside this also applies to starting any app; what I described
above is likely to make app startup slower under this common scenario.

Another situation where we want to kill background apps before sending
memory-pressure notifications is that of badly behaved apps. Imagine an
app that's misbehaved and keeps churning through memory after being sent
to the background. If we send low-memory notifications before killing
background apps we'll keep it alive indefinitely and routinely exercise
the GC. With the current setup on the other hand such an app has a high
chance of being culled by the oom killer which is a Good Thing™.

Gabriele

(*) c) is in fact still a problem, I'm working on it in bug 873284

[Thinker K.F. Li]

Gabriele Svelto <gsv...@mozilla.com> writes:

> On 16/11/2013 11:24, Andreas Gal wrote:
>> Nice analysis. Idea 1 you might be onto something. Maybe NuWa
>> rebalanced our memory equation here?
>
> Yes, it would be worth to re-evaluate our parameters for NUWA as the
> current settings were tailored before it. In addition to that, those
> parameters were also geared for 256MiB devices that had ~180MiB of
> memory available to userspace. We have a bug (886278) for
> re-evaluating them for devices with more memory.
>
> I had posted a proposal there to make those and other values
> dynamically computed depending on the total amount of free memory but
> didn't have time to follow-up on that:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=886278#c10
>
> Maybe it's time that I prioritize this stuff.
>

We are also considering to set lower adj value for the Nuwa process,
even lower than foreground. It is supposed to get more responsive of
switching Apps during low water level of memory by without relaunch Nuwa
process. It is also more efficient of memory sharing among content
processes since Nuwa was not killed, and new processes share pages with
old ones. The bad side is the unique pages occupied by Nuwa can not be
free for foreground although most pages of Nuwa are shared with
exisiting content processes.

And, we are also looking on how the b2g is on devices with even lower
memory size (< 256MB). So, maybe we should do a series of evaluations
for various configurations.

I think these values are functions of factors not only device's memory
size, but also the speed of CPU, and the number of CPU threads. I
believe LKM give processes a chance to release memory before OOM. So,
we should consider how fast the apps can be to run out of the memory,
and how fast the b2g can be to free the memory. It depends on the speed
of the CPU. So, if a device have 2GB of DRAM and 4 cores of 2GHz
processors and very fast network, it maybe not a bad idea to trigger LKM
when there is still 100MB of free memory.

This is very complicated, not easy to model it with a perfect function.
Maybe, we should do some test for various configurations, and give some
recommended values, and left this to vendors to decide and fine tuning
these values.

--
Sinker
--
天教懶漫帶疏狂